i18n versus locales

When you create a new package with its own translation domain, should you create an i18n directory or a locales directory? Basically, it does not matter in Plone 3.3 and 4.0. They both work fine. But the i18n directory is old technology and may not work anymore in Plone 5 and later. I know no downsides to the newer locales directory. So if you have the choice, you should use a locales directory. Register it like this in your configure.zcml:

Note that there is nothing magical about the locales name. You could call it Quack and it would also work, as long as you register it:

Headers and name of po file

Within a locales directory it does not matter too much what the headers look like. Some headers that you may think are vital, have absolutely no influence in practice. For example, you may have these headers:

"Language-Code: en\n"
"Language-Name: English\n"
"Domain: non.existing\n"

When those headers are in the locales/nl/LC_MESSAGES/collective.ducks.po file, then the internationalization machinery will regard the language as Dutch (nl for NetherLands as directory name) and collective.ducks as the domain as that is the name of the file minus the .po at the end. For clarity it may still be better to let the information in the headers match the folder name and file name. Update: Hanno Schlichting notes that these headers are not actually part of the gettext standard, but have been invented for the PlacelessTranslationService, so it is best to remove them from the po files, instead of trying to keep them in sync with the file name. I myself will note that i18ndude (at least the most recent 3.2 version) adds these headers by default, making it hard to keep them removed. But at least in locales directories it neither hurts to keep them, nor to remove them.

So your po file must match the pattern locales/languagecode/LC_MESSAGES/domainname.po. Note that the domain name is case sensitive. In Products.Poi the domain name is Poi, so I had to name the file Poi.po; with poi.po my translations were not picked up.

Note that if you still use an i18n directory the headers are checked. But i18n is old-style so I have not checked this too rigourously. Update: Hanno confirms that this is indeed the case; you can call your file i18n/spam.po and it will still work, as long as the headers are correct.

Domains in GenericSetup xml files

In your GenericSetup profile you can have several xml files. In some of these it makes sense to do translations. In most of those cases it only helps to use the plone domain. Let's try out most of the xml files, at least the ones contained in CMFPlone itself. Note that you are always allowed to use the plone domain, but if the xml file supports a separate domain, it is best to use that.

• actions.xml: use your own domain. Example:

  Note that when you go to the portal_actions tools in the ZMI, you will see an i18n domain specified for each action.

• catalog.xml: no i18n needed

• componentregistry.xml: no i18n needed

• contenttyperegistry.xml: no i18n needed

• controlpanel.xml: use your own domain. (Note that in Plone 3.1 you must use the plone domain.) Example:

• cssregistry.xml: no i18n needed

• diff_tool.xml: no i18n needed

• factorytool.xml: no i18n needed

• jsregistry.xml: no i18n needed

• kssregistry.xml: no i18n needed

• mailhost.xml: no i18n needed

• memberdata_properties.xml: no i18n needed

• metadata.xml: no i18n needed

• portal_atct.xml: use the plone domain. Note that this has no influence on the Collections panel in Site Setup. It is only used on the edit and criteria pages of a Collection.

• portlets.xml: use the plone domain.

• properties.xml: no i18n needed

• propertiestool.xml: no i18n needed

• rolemap.xml: no i18n needed

• skins.xml: no i18n needed

• toolset.xml: no i18n needed

• types: use your own domain

• viewlets.xml: no i18n needed

• workflows: use the plone domain

Combining i18n and locales in your own package

You could put some translations of your domain in an i18n directory and some in a locales directory. When you do that, in Plone 3 your i18n translations are ignored. In Plone 4 both are combined; I cannot imagine a sane use case for doing this though. So: for your own domain you should always use either an i18n or a locales directory.

If you want to put your own domain in a locales directory and some extra translations for the plone domain in an i18n directory, that is fine.

i18ndude

I always use i18ndude to find translations in my package and generate and update the po files. I usually copy a script with the name rebuild_i18n.sh from package A to the new package B, and adapt it to the needs of that package. It at least works on Linux and Mac and looks something like this:

#! /bin/sh

I18NDOMAIN="collective.ducks"

# Synchronise the templates and scripts with the .pot.
# All on one line normally:
i18ndude rebuild-pot --pot locales/${I18NDOMAIN}.pot \
    --merge locales/manual.pot \
    --create ${I18NDOMAIN} \
   .

# Synchronise the resulting .pot with all .po files
for po in locales/*/LC_MESSAGES/${I18NDOMAIN}.po; do
    i18ndude sync --pot locales/${I18NDOMAIN}.pot $po
done

Note the dot at the end of the rebuild-pot call to signal that i18ndude should start searching in the current directory. The merge is optional; adapt to your needs and look in the i18ndude documentation for hints.

Gotchas

If you use i18ndude to extract msgids (translatable strings) it looks for i18n:translate calls in templates and xml files; it looks for the corresponding i18n:domain specification to check if this msgid belongs to the domain you are interested in (specified in the --create option). In python files i18ndude simply looks for all strings wrapped in an underscore function: _('My translatable string'). This function is usually defined like this:

from zope.i18nmessageid import MessageFactory
_ = MessageFactory('collective.ducks')

The gotcha now is that i18ndude is not smart enough to know which domain the string belongs to. It will report all 'underscored' strings, whether you look for your own domain or the plone domain. Usually you are only interested in your own domain. If you still need to use the plone domain in a python file, it is best to make sure this is not done with an underscore. For example, when customizing the folder_copy.cpy script of standard Plone in an own skin layer, I usually change it from this:

from Products.CMFPlone import PloneMessageFactory as _
message = _(u'One or more items not copyable.') 

to this:

from Products.CMFPlone import PloneMessageFactory as PMF
message = PMF(u'One or more items not copyable.')

That way, i18ndude will not detect that the string 'One or more items not copyable.' is translatable, so it will not end up in your pot and po files, so the existing translation from plone.app.locales is used.

Special case: translate the profile title and description

Say you have a configure.zcml like this:

i18ndude will not pick up the title and description, so you will have to add it manually. Create a locales/manual.pot file with something like this:

# --- PLEASE EDIT THE LINES BELOW CORRECTLY ---
# SOME DESCRIPTIVE TITLE.
# FIRST AUTHOR , YEAR.
msgid ""
msgstr ""
"Project-Id-Version: collective.duck 1.0\n"
"POT-Creation-Date: 2010-09-01 09:58+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
"Last-Translator: FULL NAME \n"
"Language-Team: LANGUAGE \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=1; plural=0\n"
"Language-Code: en\n"
"Language-Name: English\n"
"Preferred-Encodings: utf-8 latin1\n"
"Domain: collective.ducks\n"

#: Profile title in configure.zcml
msgid "Collect All Ducks"
msgstr ""

#: Profile description in configure.zcml
msgid "Ducks of all nations: unite!"
msgstr ""

Rebuild the pot file and merge the manual pot file in it, something like this:

i18ndude rebuild-pot --pot locales/collective.ducks.pot --merge locales/manual.pot --create collective.ducks .

Synchronize the pot file with the po files (we will just do the Dutch file here):

i18ndude sync --pot locales/collective.ducks.pot locales/nl/LC_MESSAGES/collective.ducks.po

Add the translations in your po file, restart your zope instance and the translations should be visible in the Add-ons panel in the Site Setup (and the portal_quickinstaller in the ZMI).

Extra translations for an existing domain

This section is true for adding extra translations (new msgids) to any domain, but the plone domain is of course the one for which this is most often needed.

In Plone 3, the standard plone domain translations are done in an i18n directory. If you create a locales directory with extra plone translations, these will cancel all existing i18n translations for plone, so this is not good. You should create an i18n directory for this.

In Plone 4 you should add extra plone domain translations in a locales dir.

In general, you should follow the lead of the standard translations for the domain you want to add msgids for. If they are in an i18n directory, you should add your's in an i18n directory. If they are in a locales directory, you should add your's in a locales directory.

Note that in Plone 4 it looks like first all locales are loaded for all packages, and then all i18n directories are loaded. This might depend on the order in which the zcml for the packages is loaded though.

Overriding translations

In Plone 3.3 you can override translations that are in the i18n directory of a package (for example plone.app.locales, which is the successor or Products.PloneTranslations). The best way to make sure your translations are picked over the default ones, is by creating an i18n directory within your zope instance. Easiest is to use collective.recipe.i18noverrides for this.

Note that you can not use an i18n directory in the instance to override translations from a locales directory.

In Plone 4 the i18n directory of the instance is completely and utterly ignored. What you can do is override some translations of a locales directory. You can create a new plone package and use that only to override some translations, or you add something to an existing package. Let's say you want to override the "You are here:" translation from the bread crumbs (path bar). In Dutch it says "U bent hier:". We now want it to say: "U bent hier naartoegevlogen:". You locate the plone.po file for your translation and copy it to your locales directory. In our case we have created a package called customer.translations so copy this file:

plone.app.locales-4.0.0-py2.6.egg/plone/app/locales/locales/nl/LC_MESSAGES/plone.po

to this file:

customer.translations/customer/translations/locales/nl/LC_MESSAGES/plone.po 

Then remove whatever translations you do not need to override, and do your change, so you end up with a po file with some headers and this translation:

#. Default: "You are here:"
#: plone.app.layout/plone/app/layout/viewlets/path_bar.pt:6
msgid "you_are_here"
msgstr "U bent hiernaartoe gevlogen:"

In your buildout.cfg make sure this package is added to the eggs. The tricky thing now is to make sure that the zcml for this package is loaded before the zcml of other packages; that way our translations win. The way to do this, is to list it as the first (and possibly only) item in the zcml option, so something like this:

[instance]
...
eggs =
    Zope2
    Plone
    ${buildout:eggs}
    customer.translations
zcml =
    customer.translations

Warning: this works in Plone 3 too, but it has a problem. In Plone 3 the translations for the plone domain are in an i18n directory. If we add a plone.po file in our locales directory, the effect is that all other translations from the plone domain are lost. In other words: if you override an i18n folder with your own locales folder, all translations for that domain that are not in your po file, are lost. So you would have to copy over the complete original po file. In that case it is probably easier to use the already mentioned collective.recipe.i18noverrides. To be clear: this is true when using Plone 3.

To summarize this part:

• If you want to override translations from an i18n directory:
  • On Plone 3.3: use collective.recipe.i18noverrides.
  • On Plone 4: copy (part of) the po file to the locales directory of your own package.
• If you want to override translations from a locales directory it is the same on Plone 3.3 and 4: copy the relevant part of the po file to the locales directory of your own package. Copying the entire po file works as well of course, but is not needed. Also you need to make sure the zcml of your package is loaded first. On Plone 3.3 it seemed less reliable, but that may be because I was meanwhile also combining i18n and locales directories for the same domain, which is not a good idea.

Restrict the loaded languages

(Update: includes info from Hanno about zope_i18n_allowed_languages.)

To speed up zope startup time and use less memory, you can set an environment variable to restrict the languages for which po files are loaded. To restrict them to English, Dutch and German, give this a value of en, nl, de. The commas are optional. In buildout.cfg this would be something like this:

[instance]
...
environment-vars =
  PTS_LANGUAGES en nl de
  zope_i18n_allowed_languages en nl de
  zope_i18n_compile_mo_files true

The zope_i18n_compile_mo_files setting is optional, see the next section. But why do we specify the allowed languages twice? Some words from Hanno, who pointed me to zope_i18n_allowed_languages:

In Plone 3.3 there is only PTS_LANGUAGES and it affects both i18n and locales folders. In Plone 4 there is also zope_i18n_allowed_languages. The new one now affects locales folders and the old one only affects i18n folders. So to get the full memory saving affect of not loading too many translation files, you need to specify both on Plone 4. Specifying the new one in Plone 3.3 does not hurt, it just does not do anything.

Compiled translation files

.po files need to be compiled before Plone can do something with them. When you compile a domain.po file, you get a domain.mo file. This is the file that is read by the translation machinery when looking up a translation for a msgid. To compile a file manually, you need the msgfmt program:

msgfmt -o domain.mo domain.po

If your package contains po files, Plone 3.3 compiles them when starting up your instance. In Plone 4 this is optional. For example, you might want to skip this when you do not need any translations at all in your site. Actually, it is skipped by default. To enable the compile step, add the following to the instance or zeoclient part of your Plone 4 buildout.cfg:

environment-vars =
    zope_i18n_compile_mo_files true

Note that the value (true in this case) does not matter: the relevant code in zope.i18n simply looks for the existence of the variable and does not care what its value is.

In Plone 3, any existing mo files in an i18n directory of a package are ignored. Instead, when the Zope instance starts up, all po files are compiled automatically and put in a directory like var/instance/pts.

Also in Plone 3, any po files in a locales directory are compiled inside that same locales directory: locales/nl/LC_MESSAGES/domain.po is compiled to a file locales/nl/LC_MESSAGES/domain.mo.

In Plone 4, the var/instance/pts directory is not used at all. All po files in an i18n or locales directory are compiled inside that same directory.

On startup, Zope (actually the PlacelessTranslationService) may see that a po file already has an accompanying mo file in the correct directory. It then compares the last modification date of the two files. If the po file is more recent, then a new mo file is compiled.

Note that you should not put the compiled mo files in subversion (or another version control system) as they can just be automatically created. Once you release a package to the public, putting the mo files in the released source could be handy though: if your package without mo files is installed by for example the root user in a directory where user zope has read access but not write access, then user zope will get an error when starting Plone. plone.app.locales does this correctly. I have not done this myself yet (and some of my packages probably still contain mo files in subversion) so I will not further comment on how to do this. I do want to add support for this in zest.releaser or in an optional support package for that.

If you want to do it manually, this shell script does the trick (at least when you have proper bash, find and msgfmt commands available):

for po in $(find . -path '*/LC_MESSAGES/*.po'); do
    msgfmt -o ${po/%po/mo} $po;
done

Plus you will need to make sure mo files are included in the created source distribution by creating a MANIFEST.in file next to your setup.py file; this works for me:

recursive-include collective *
global-exclude *pyc

Update: Okay, I have just created and released http://pypi.python.org/pypi/zest.pocompile for this, for use in combination with zest.releaser or as stand-alone command line tool.

Conclusions

• When you create a package with a new translation domain: use locales.
• When extending existing translations, follow the lead of the package you are extending to determine whether you should use i18n or locales.
• When overriding existing translations, also follow the lead of the package you are overriding. Make sure the zcml of your package is loaded before that of the other package. On Plone 3.3 with an i18n dir: use collective.recipe.i18noverrides.
• When using locales, make sure your po file matches the pattern locales/languagecode/LC_MESSAGES/domainname.po and note that the domain name is case sensitive.
• In GenericSetup files, use your own domain for:
  • actions.xml
  • controlpanel.xml
  • types
• In GenericSetup files, use the plone domain for:
  • portal_atct.xml
  • portlets.xml
  • workflows

Have fun translating!

Okay, so what is new?

Plone 4 is a lot faster. We focused on that. We tested plips for their effect on speed. Hanno Schlichting was the hero here. These are long term improvements, certainly also in the infrastructure. We use Zope 2.12 and python 2.6, which helps reduce memory usage. Plone 4 is about 50 percent faster than Plone 3. Compared to other open source CMSes (all out of the box, without tweaking) we perform very good as well. We are faster than Joomla, Drupal and Wordpress when you look at requests per second.

BLOBs: Binary Large OBjects. Large files are no longer stored in the ZODB (Zope Object DataBase) but in blob storage. This means less memory usage, less database growth, less pushing small objects out of the cache. On one site we saw the memory consumption go from 14 GB to 3 GB.

We got a new visual editor: TinyMCE. Kupu has shipped with Plone since 2005. It has wonderful features and was good at the time. We were happy with it, but it was hard to maintain a visual editor that mostly only we were using. Instead TinyMCE is an existing editor maintained by others. So we just need to take care of integration in Plone, which Rob Gietema of Four Digits has done a lot of work for. It also works on Plone 3. Now it is part of Plone core. Future development will be based on TinyMCE. It has better image upload, table editing, inserting hyperlinks, etcetera. Probably this is the most important user interface improvement in Plone 4.

Plone 4 has a new look. If you have stared at the green and blue boxes for ten years, you will be happy with the new theme. It is a refreshing change. A grid based theme. The old theme has been improved over many years; the new one is younger and smaller, so may be missing some fixes. The old theme is still available, so that should make the upgrade experience of a theme from Plone 3 to 4 much smoother. There is also a basic theme without actual theming, which can work nicely as basis for an own theme.

JQuery Tools is shipped and used. We use it for 'modal dialogs', popup-like dialogs for those times when loading a complete new page makes no sense.

User management has improved. Full names are used everywhere. You can login with email addresses, when you turn that on. Defining your own member data is more flexible. You can assign portlets to groups.

We have full text indexing for Eastern languages. For Western developers that is very hard to do. It makes Plone usable for half a billion people more than before.

When you startup Zope for the first time without a Plone Site, you are much friendlier greeted. We should have done that much sooner. Hanno has done this, probably in half an hour. :-)

It is probably the simplest upgrade of any Plone upgrade ever. It is a big version, do your backups and stuff, but you should be fine, at least when upgrading from any Plone 3 version. Some work is being done do make it possible to export for example a Plone 2.1 site and import it in a fresh Plone 4 site, if you run into problems.

What is next?

Upgrading an add-on package from Plone 3 to Plone 4 is not much work. If you need this for an add-on, just send a friendly email to the maintainer.

Plone 4.0.1 is expected to be released tomorrow, 4.0.2 at the end of September, then each month. Usually I advise to wait for the first bugfix release, but for Plone 4 I don't have the feeling that is necessary; but 4.0.1 is there soon anyway.

Now about the mid-term, 2010-2011. Plone 4.1 probably early next year. Expected: Amberjack guided tours of the user interface, improved commenting, new collections (renaming them did not help, so we did some heavy restructuring to make them lighter), SiteAdmin role (basically Manager without ZMI access), password policies.

Plone 4.2: we don't know how far away this is. Chameleon template engine for faster rendering (probably about 20% speed improvement), architecture preparation for Deco, content governance (which person is responsible for maintaining it, is it still up to date), batch editing.

Plone 4.3 may not even happen, we may be close enough to Plone 5 by that time.

Somewhere on the horizon is the magical Plone 5. It will improve the user interface for editing and put us at the front of the pack there, using Deco and tiles, unified content types, new editing interface.

Remark from the room: it would be good to get a performance comparison between Sharepoint 2010 en Plone 4.

Note: if you are Dutch you may want to go to the Dutch version of my weblog to see some extra entries (at time of writing only one, but there will be more) in Dutch about this user day.