International emails

i18n/l10n

Two terms widely used are:

internationalization
i       18         n

localization
l    10    n

Roughly said, in a Plone context, internationalization is making sure the content or the UI is translated into several languages. Localization is making sure that 3 May 2009 is 05-03-2009 in the USA and 03.05.2009 in Germany.

These two terms are not really the focus here though. The point is: how do you make sure that an email sent from Plone (or any python application really, if you ignore some details) with a Chinese name as From address, a Japanese name as To address, a Russian Subject and a Korean body text is delivered without errors.

Now do not think: "I live and work in America, I only need ascii." Don't you have Spanish colleagues? Some friends from your year abroad at that French university? A few Chinese clients? You could use only ascii, but you might regret that:

utf-8 is not unicode

Repeat after me: "utf-8 is not unicode", "utf-8 is not unicode", "utf-8 is not unicode":

>>> type('ascii')

>>> type('utf-8')

>>> type(u'unicode')

Basics

Sending an email in Plone goes something like this:

charset = portal.getProperty(
  'email_charset', 'ISO-8859-1')
mailhost = getToolByName(portal, 'MailHost')

mailHost.send(message = msg,
              mto = address,
              mfrom = mfrom,
              subject = subject,
              charset = charset)

What can go wrong with that?

• Hard to read headers:

  From: RenXX Artois
  

• Hard to read body text:

  lettere accentate: ò ùâ
  

• Unrecognized addresses:

  To: undisclosed recipients
  

• No email body: C

• UnicodeDecodeErrors/UnicodeEncodeErrors

Some examples can be found in Poi issues 146 and 161.

Parsing/formatting addresses

The To and From fields should have something like this:

Maurits van Rees 

The standard python email package has nice utilities for this:

>>> from email.Utils import parseaddr
>>> from email.Utils import formataddr
>>> formataddr(('Maurits van Rees',
                'maurits@example.org'))
'Maurits van Rees '
>>> parseaddr(
      'Maurits van Rees ')
('Maurits van Rees', 'maurits@example.org')

These functions can get confused by strange characters. You can guard against that by parsing the address that you have just formatted and seeing if the parsed information still makes sense:

from_address = portal.getProperty(
  'email_from_address', '')
from_name = portal.getProperty(
  'email_from_name', '')
mfrom = formataddr((from_name, from_address))
if parseaddr(mfrom)[1] != from_address:
    # formataddr probably got confused
    # by special characters.
    mfrom = from_address

Character sets

The python email.Charset module has interesting information about how email headers and body text should be encoded depending on the input character set. Some examples (QP is quoted printable):

input         header enc  body enc  output conv
iso-8859-1:   QP          QP        None 
iso-8859-15:  QP          QP        None 
windows-1252: QP          QP        None 
us-ascii:     None        None      None 
big5:         BASE64      BASE64    None 
euc-jp:       BASE64      None      iso-2022-jp 
iso-2022-jp:  BASE64      None      None 
utf-8:        SHORTEST    BASE64    utf-8 
...

If that does not make sense, perhaps this helps:

This information is used when creating email headers:

>>> from email.Charset import Charset
>>> latin = Charset('iso-8859-1')
>>> utf = Charset('utf-8')
>>> latin.header_encode('René Artois')
u'=?iso-8859-1?q?Ren=C3=A9_Artois?='
>>> utf.header_encode('René Artois')
'=?utf-8?q?Ren=C3=A9_Artois?='

and encoding body text:

>>> latin.get_body_encoding()
'quoted-printable'
>>> latin.body_encode('René Artois')
'Ren=C3=A9 Artois'
>>> utf.get_body_encoding()
'base64'
>>> utf.body_encode('René Artois')
'UmVuw6kgQXJ0b2lz\n'

This may look confusing. Surely if you get an email with a text or subject like this it is unreadable? No, your email program should be smart enough to display this to you in a readable fashion. No need for the funny face:

Formatting headers

Instead of using email.Charset for formatting headers you normally use the email.Header module:

>>> from email.Header import Header
>>> subject = 'Re: René'.decode('latin-1')
>>> subject
u'Re: Ren\xc3\xa9'
>>> subject = Header(subject, 'latin-1')
>>> subject

>>> print subject
=?iso-8859-1?q?Re=3A_Ren=C3=A9?=

Formatting the body

You will need to know which character set the body text has, or at least in which character set it can be encoded without errors. This snipped tries three character sets:

charset = portal.getProperty(
  'email_charset', 'ISO-8859-1')
for body_charset in 'US-ASCII', charset, 'UTF-8':
    try:
        message = message.encode(body_charset)
    except UnicodeError:
        pass
    else:
        break

If the message only contains ascii characters, then at the end of this snippet the message is encoded in ascii and the body_charset variable is 'US-ASCII'.

Send it

We have done all the hard work with the Headers so now we can use the 'send' method:

# Create the message.
# 'plain' stands for Content-Type: text/plain
from email.MIMEText import MIMEText
msg = MIMEText(message, 'plain', body_charset)
msg['From'] = email_from
msg['To'] = email_to
msg['Subject'] = subject
msg = msg.as_string()
mailhost = getToolByName(portal, 'MailHost')
mailhost.send(message=msg)

Using secureSend

Easier is to use the secureSend method; using with the Header class is not needed then, as secureSend takes care of that:

email_msg = MIMEText(message, 'plain', body_charset)
mailhost.secureSend(
  message = email_msg,
  mto = email_to,
  mfrom = email_from,
  subject = subject,
  charset = header_charset)

Now international email sending should work:

Images courtesy of:

• http://engrishfunny.com/
• http://punditkitchen.com/
• http://icanhascheezburger.com/
• http://www.alloallo.yoyo.pl/