Stani: Phatch - Batch photo processing for everyone

Lots of photo data, including metadata. You have ImageMagick. Phatch allows you to resize images, do colour profiling. Combine actions. Prefabricated recipes. You don't need to know how to do graphics manipulation. You can remove metadata so you can upload photos anonymously to sites. You can have a droplet on your desktop; just drop an image on it and it will do the actions you specified. Safe and unsafe mode. Safe restricts you; in unsafe mode you have the full power of python. It's a command line tool, but also a UI. Uses open source fonts. Phatch can do anything Gimp and ImageMagick does and more as they are the back ends, including PIL and blender. Its goal is to unite all open source graphics tools. Anyone can submit new objects. Works on Linux and mostly on Windows and Mac, but we need more developers there. About six developers now. Someone has plans to work on Django integration. Link: http://photobatch.stani.be/

Jasper Spaans, Fox-IT: Pycuda - massively parallel computing

Calculating with the GPU (Graphical Processing Units). GPU's are incredibly fast for parallel computations compared to their price. Cuda is only nor Nvidia. OpenCL does that too, and that can also use ATI/AMD. Use it for parallel algorithms. PyCuda is a python interface for cuda. Slides in Dutch: http://jasper.es/talks/pun2009-09

Wim Feijen, Go2People: Bazaar (and mercurial and git)

Bazaar: you can have one or more repositories. You can work and commit locally. It's simple. It tracks movements. You can branch projects. Look at http://github.com (not only for git users) and http://bitbucket.org, for free project hosting.

Remco Wendt, Maykin Media: libraries that you love

Does anyone know of a library that is cool to use? Speak up about it!

• Simon: greenlets; does micro threads; nice for animation, xml parser, http://pypi.python.org/pypi/greenlet
• zc.buildout: separate different project environments, repeatable deployment, http://pypi.python.org/pypi/zc.buildout
• pysvn: bindings for svn in python, http://pysvn.tigris.org/
• Ivo: virtualenv, keep libraries separate, sandbox them, http://pypi.python.org/pypi/virtualenv
• Tim: Paste: WSGI library (and other stuff), http://pypi.python.org/pypi/Paste
• werkzeug: like paste but different - http://pypi.python.org/pypi/Werkzeug
• WebOb, request and response objects: http://pythonpaste.org/webob/
• repoze.who, repoze.what: WSGI authentication and authentication framework, middleware, http://docs.repoze.org/who/ and http://what.repoze.org/docs/1.x/
• Fabric: remote deployment tool, http://docs.fabfile.org/0.9/
• grok: web framework, zope 3, without zcml headaches, http://grok.zope.org/
• scrapy: build web scrapers, http://scrapy.org/
• lamson: smtp server framework in python, http://lamsonproject.org/
• pep8, python style guide checker, finally easy_installable, http://pypi.python.org/pypi/pep8

Jan Jaap Driessen (Health Agency) and Sylvain Viollon (Infrae): Grok

Web framework, based on Zope. Even cavemen can use Zope. Just had a sprint about Grok. Takes the pain out of working with Zope. Zope has lots of layers, configuration, zcml, adapters, whatever. With Grok you do not register your stuff in zcml. Set of default settings. Grok "groks" your code, understands your code without needing a separate configuration language. Grok 1.0 is due for release this or next week. We have been using it for three years already. http://grok.zope.org

Gijs Molenaar, UvA, CWI: MonetDB and python 3

DBMS, developed by CWI, open source, BSD, platform for scientific research, not only SQL, http://www.monetdb.nl. Good for big databases; SkyServer is 6TB. For query intensive apps that are mostly read-only. For complex data models beyond SQL. Sometimes outperforms other databases by a hundred times for big tables.

It is a column based storage database. All columns are shown separately. Flexible kernel, which is easily extensible. SQL front end, but not limited to SQL; can use Xquery.

You can store xml documents in your database. Monet takes the document apart and stores it in several tables.

There are APIs for Ruby, PHP, python, perl (needs a rewrite but nobody wants to touch it), C, java. The CWI at the University of Amsterdam wanted me to rewrite the python API. MonetDB API is the wire protocol. Basic functions: set up connection, execute commands, parse the response, chop queries and responses in blocks. python.sql is the implementation of all DBAPI 2.0 functionality. Bridge between this and MonetAPI (mapi).

There were tests available for the MySQL python API, so I borrowed those for Monet. We looked at supporting python 2 and 3. We did not want to use two trees. We decided to do a 2+3 hybrid, though Guido van Rossum tells us not to do that. In python 3 there is a new IO layer with bytes and strings. New IO layer is used in the mapi layer. We made two versions here, mapi2.py and mapi3.py as it was too hard to combine, with exceptions that were no longer there in python 3. We do a python 3 version check in only two places, like conditionally importing mapi2 or mapi3, so not too bad. Tip: don't use print, use logging.

The hybrid python2-3 model works quite well in our case as the API is quite small. We minimised code duplication. Not advised for all projects.

Future work: bug fixes, port the xquery api, currently I am rewriting the C api (mapilite), unit tests in python using ctypes, having problem with structs.

Something completely different. http://pythonic.nl non-profit amateur web hosting provider for python web development. Focus on django.

Question: How mature is MonetDB? Well, there have been cases of data loss, but it has gotten better the last few years and it has been used in various projects.

Slides: http://gijs.pythonic.nl/static/pun-090924.pdf

Tim Molendijk: debugging

I usually just hunt for a bug, kill it and forget how I did the debugging and not improve it next time. But I am learning. I should probably use the python debugger, but usually I start with adding a few print statements. But with WSGI applications that normally does not work. So yes, next step is (i)pdb. For Django: Django debug toolbar.

Suggestions from the crowd: post mortem debugger, Twill, zc.testbrowser, mechanize, Selenium, winpdb, pydev for eclipse (you can still use vim for writing the code if you have to), werkzeug (for django).

Maurits van Rees: international emails

Short version of a presentation I gave at my employer Zest Software and that I blogged about recently. Suggestion from Jasper Spaans: use lamson (mentioned above), an smtp server framework in python.

i18n/l10n

Two terms widely used are:

internationalization
i       18         n

localization
l    10    n

Roughly said, in a Plone context, internationalization is making sure the content or the UI is translated into several languages. Localization is making sure that 3 May 2009 is 05-03-2009 in the USA and 03.05.2009 in Germany.

These two terms are not really the focus here though. The point is: how do you make sure that an email sent from Plone (or any python application really, if you ignore some details) with a Chinese name as From address, a Japanese name as To address, a Russian Subject and a Korean body text is delivered without errors.

Now do not think: "I live and work in America, I only need ascii." Don't you have Spanish colleagues? Some friends from your year abroad at that French university? A few Chinese clients? You could use only ascii, but you might regret that:

utf-8 is not unicode

Repeat after me: "utf-8 is not unicode", "utf-8 is not unicode", "utf-8 is not unicode":

>>> type('ascii')

>>> type('utf-8')

>>> type(u'unicode')

Basics

Sending an email in Plone goes something like this:

charset = portal.getProperty(
  'email_charset', 'ISO-8859-1')
mailhost = getToolByName(portal, 'MailHost')

mailHost.send(message = msg,
              mto = address,
              mfrom = mfrom,
              subject = subject,
              charset = charset)

What can go wrong with that?

• Hard to read headers:

  From: RenXX Artois
  

• Hard to read body text:

  lettere accentate: ò ùâ
  

• Unrecognized addresses:

  To: undisclosed recipients
  

• No email body: C

• UnicodeDecodeErrors/UnicodeEncodeErrors

Some examples can be found in Poi issues 146 and 161.

Parsing/formatting addresses

The To and From fields should have something like this:

Maurits van Rees 

The standard python email package has nice utilities for this:

>>> from email.Utils import parseaddr
>>> from email.Utils import formataddr
>>> formataddr(('Maurits van Rees',
                'maurits@example.org'))
'Maurits van Rees '
>>> parseaddr(
      'Maurits van Rees ')
('Maurits van Rees', 'maurits@example.org')

These functions can get confused by strange characters. You can guard against that by parsing the address that you have just formatted and seeing if the parsed information still makes sense:

from_address = portal.getProperty(
  'email_from_address', '')
from_name = portal.getProperty(
  'email_from_name', '')
mfrom = formataddr((from_name, from_address))
if parseaddr(mfrom)[1] != from_address:
    # formataddr probably got confused
    # by special characters.
    mfrom = from_address

Character sets

The python email.Charset module has interesting information about how email headers and body text should be encoded depending on the input character set. Some examples (QP is quoted printable):

input         header enc  body enc  output conv
iso-8859-1:   QP          QP        None 
iso-8859-15:  QP          QP        None 
windows-1252: QP          QP        None 
us-ascii:     None        None      None 
big5:         BASE64      BASE64    None 
euc-jp:       BASE64      None      iso-2022-jp 
iso-2022-jp:  BASE64      None      None 
utf-8:        SHORTEST    BASE64    utf-8 
...

If that does not make sense, perhaps this helps:

This information is used when creating email headers:

>>> from email.Charset import Charset
>>> latin = Charset('iso-8859-1')
>>> utf = Charset('utf-8')
>>> latin.header_encode('René Artois')
u'=?iso-8859-1?q?Ren=C3=A9_Artois?='
>>> utf.header_encode('René Artois')
'=?utf-8?q?Ren=C3=A9_Artois?='

and encoding body text:

>>> latin.get_body_encoding()
'quoted-printable'
>>> latin.body_encode('René Artois')
'Ren=C3=A9 Artois'
>>> utf.get_body_encoding()
'base64'
>>> utf.body_encode('René Artois')
'UmVuw6kgQXJ0b2lz\n'

This may look confusing. Surely if you get an email with a text or subject like this it is unreadable? No, your email program should be smart enough to display this to you in a readable fashion. No need for the funny face:

Formatting headers

Instead of using email.Charset for formatting headers you normally use the email.Header module:

>>> from email.Header import Header
>>> subject = 'Re: René'.decode('latin-1')
>>> subject
u'Re: Ren\xc3\xa9'
>>> subject = Header(subject, 'latin-1')
>>> subject

>>> print subject
=?iso-8859-1?q?Re=3A_Ren=C3=A9?=

Formatting the body

You will need to know which character set the body text has, or at least in which character set it can be encoded without errors. This snipped tries three character sets:

charset = portal.getProperty(
  'email_charset', 'ISO-8859-1')
for body_charset in 'US-ASCII', charset, 'UTF-8':
    try:
        message = message.encode(body_charset)
    except UnicodeError:
        pass
    else:
        break

If the message only contains ascii characters, then at the end of this snippet the message is encoded in ascii and the body_charset variable is 'US-ASCII'.

Send it

We have done all the hard work with the Headers so now we can use the 'send' method:

# Create the message.
# 'plain' stands for Content-Type: text/plain
from email.MIMEText import MIMEText
msg = MIMEText(message, 'plain', body_charset)
msg['From'] = email_from
msg['To'] = email_to
msg['Subject'] = subject
msg = msg.as_string()
mailhost = getToolByName(portal, 'MailHost')
mailhost.send(message=msg)

Using secureSend

Easier is to use the secureSend method; using with the Header class is not needed then, as secureSend takes care of that:

email_msg = MIMEText(message, 'plain', body_charset)
mailhost.secureSend(
  message = email_msg,
  mto = email_to,
  mfrom = email_from,
  subject = subject,
  charset = header_charset)

Now international email sending should work:

Images courtesy of:

• http://engrishfunny.com/
• http://punditkitchen.com/
• http://icanhascheezburger.com/
• http://www.alloallo.yoyo.pl/

In all versions:

• The GS Profile name is shown, otherwise the package name is shown.
• If the getProductVersion method of the QI cannot find a version, no version is shown.
• Packages must be in the Products name space or use the five:registerPackage directive in zcml, otherwise they are not shown in the Product Management section of the Zope Control Panel.
• For getting the version number from the version.txt file, the package must be listed in that Product Management section. For getting the version number from setup.py or from the GS profile (the metadata.xml file) this is not necessary.
• Packages that rely on an external method (so no GS profile) for getting installed and that are not in the Product Management section, are not listed in the QI.
• Note that the Products section of the Zope Control Panel only looks for version.txt, at least in Zope 2.10.

Plone 3.0

• Plone 3.0.6 uses QI 2.0.4.
• getProductVersion gets the version from version.txt only.
• The GS Profile version is never shown.
• Packages must be in the Products name space or use the five:registerPackage directive, otherwise they are not shown in the QI. Only with this QI version is this true both for packages with an external method and for packages with a GS profile.

Plone 3.1/3.2

• Plone 3.1.7 and 3.2.2 use QI 2.1.6.
• getProductVersion gets the version from version.txt; when that fails it gets the version from the GS profile.
• Packages with a GS profile are always shown in the QI, also when they do not use five:registerPackage and are not in the Products name space.
• When a GS profile version is found, this version is shown, except that when five:registerPackage is used (or the Products name space) and there is a version.txt, then that version is shown.

Plone 3.3

• Plone 3.3rc2 uses QI 2.1.7.
• getProductVersion gets the version from setup.py; else it gets the version from version.txt; it specifically does not get the version from the GS profile.
• setup.cfg is also used, so if you have tag_build = dev in the egg_info section, the version will have 'dev' behind it.

Conclusion

It looks like the only way to have the same version number listed in all Plone 3.x versions is with version.txt. Follow these guidelines and (at least for the studied QI versions) the version number from version.txt is always shown:

• Use the Products name space or use the five:registerPackage directive. (Tip: when using paster to create your package, answer yes to the question 'Are you creating a Zope 2 Product?')
• Use a version.txt.
• In the setup.py use the version number from version.txt as the package version.

With this strategy it is possible to use GS profile versions that are entirely different from the version of your package, without confusing users. Specifically, I think I will start using just integers for the GS profile versions, instead of the dotted numbers I mostly use now.

One last conclusion: for Plone 3.0 (QI 2.0) remember that if you only have a GS profile and no external install method, your package will not be shown in the QI so you will have to use the portal_setup tool instead to install your package.