I made a report for school, about what I did the past year (well, about nine months) at Zest Software. You can download the report (pdf). Read on for the preface and abstract of the report.

Preface

This report spans the period of May 2006 till January 2007. During that time, I was working for Zest Software in Hoogvliet, The Netherlands. This was part of the "dual route" of my study Informatics at the Rotterdam Institute for Informatics Studies (RIVIO) of the Hogeschool Rotterdam. I worked four days a week for Zest, and studied on Fridays.

Zest Software is a Plone company, so my work there is mostly in Plone and in Zope and Python as underlying technologies. But other subjects pop to the surface from time to time, like subversion, LDAP and Linux administration in general.

The main goal of writing this report is to convince Hans Manni from the Hogeschool Rotterdam that I learned a lot and worked hard these past months at Zest Software and that I should receive a good number of study points. But I like to give some extra zest to this report by making it relevant for a bigger audience, specifically Plone developers. I will therefore focus on three subjects that I think give a broad overview of what I have been doing and are also interesting to the general Plone community.

Just like in my last report, I can say that I have learned and laughed a lot at Zest Software and I thank my colleagues for that. I look forward to working on my final study assignment with them.

Abstract

After the introduction, the first subject is subversion. Importing your product in the collective shows how to move the code of your product from your own subversion repository to the Plone collective (or any other repository) including all the history.

In ldapconfig: connecting Plone and LDAP we take on a subject that is always good for a lot of questions on Plone mailing lists. Connecting with LDAP (or Microsoft Active Directory) is certainly doable, but there are some gotchas. To make it easier for us and for Plone developers in general to get this to work, we made ldapconfig, a product that tries to reduce the complexity of this subject to a relatively simple configuration file. This product is presented in the third chapter.

Last but not least is a tutorial on Zope 3 technologies: Embrace and Extend: The Zope 3 Way. Slowly but surely Zope 2 is being replaced by the newer and cleaner Zope 3. This makes it possible to extend existing Zope (or Plone) products in a very clean, non-intrusive way, that is future-proof.

By personal convention and as service to the reader, I have included the most important files that I created for this report itself in the appendices.

• I had some trouble upgrading my desktop as I had the opera browser package installed (not supported by Ubuntu). Ubuntu wanted to change some directory into a symlink, but an Opera file was in the way. Solution: uninstall opera before you start upgrading (and install it again later if you want):

    dpkg --remove opera
  

• I had previously installed firefox-dom-inspector. Somehow the upgrade didn't go so well for that part. In Firefox, in the Extra-Addons menu item, the Dom Inspector was listed as broken. Reinstalling it solved the problem:

    aptitude reinstall firefox-dom-inspector
  

• The fonts in my Emacs editor looked a bit ugly or in the case of my laptop they were simply boxes instead of characters. The cause is that fonts were moved to a different location. So you have to manually edit your /etc/X11/xorg.conf file and change lines like:

    FontPath  /usr/share/X11/fonts/misc
    into:
    FontPath  /usr/share/fonts/X11/misc
  

Those were the three problems I had. For the rest it went smooth as always. And it all seems to work just fine. Kudos to the Ubuntu community!

Oh, of course when you have compiled and installed some programs yourself, you may need to do that again. For instance I needed to reinstall some Zope versions, as the standard Ubuntu python version has been updated to 2.4.4. This version is not expected by e.g. Zope 2.9.5; it wants 2.4.2 or 2.4.3. This should not be a problem I guess, but it does mean that you need to do:

    ./configure --with-python=/usr/bin/python

That makes the configure script accept your python version. Then I tried to make Zope, but this gave lots of errors. This was solved by installing some development packages:

    aptitude install build-essential

First I noted this blog entry by Tom Lazar. Two simple lines in a .pdbrc file in your home directory are enough to give you tab completion in your pdb session:

    import rlcompleter
    pdb.Pdb.complete = rlcompleter.Completer(locals()).complete

Be sure to read the comments at Toms site. Also see the original recipe at ASPN.

I tried this and it worked perfectly. That should give me a productivity boost. Since I work with Zope I started a bin/zopectl debug session and tried tab completion there, but that didn't work. That's logical as the zope debug is not the same as a pdb session. Daniel Nouri came with the obvious answer: just do import pdb; pdb.set_trace() in your debug session! Works fine then.

But then I thought: if this is possible in the pdb, then it should surely be possible in a normal python session. So I searched and found an article by Jacob on ParTecs Weblog. This shows that you can add a .pythonrc.py file in your home directory which contains:

    import rlcompleter, readline
    readline.parse_and_bind('tab: complete')

Set an environment variable:

    export PYTHONSTARTUP=~/.pythonrc.py

Now startup python and you have tab completion. Grand! Jacob's article shows another handy function that you can put in that file, so be sure to read it. While you are at it, check out some more pdbrc ideas. I added some lines there as well.

This still doesn't give you tab completion in a zopectl session though, unless you do the pdb trick. But I prefer not to type/paste that everytime I do some zopectl debugging. I guess zopectl doesn't even try loading that .pythonrc.py file, probably for safety reasons, so noone can sneak in some code. Let's see if we can find a workaround.

Hm, the $PYTHONSTARTUP environment variable is only read for interactive python shells. Apparently a debug session is not considered interactive.

Okay, for starters I can shave off a few key strokes. Run bin/zopectl debug and then:

    import user

This will run your ~/.pythonrc.py file. Two notes:

• This does not use the $PYTHONSTART environment variable. It is hardcoded to load only the .pythonrc.py file in your home directory. So if you can't or don't want to set that environment variable, you can still just do an import user in an interactive python session and your .pythonrc.py file will get loaded.
• If you have defined a function in your file, it is not available for use directly like in a really interactive python session, but only with the module name attached; so with the mentioned example file, you can need to type user.source(user), which shows the source of the user module.

Now my reasoning goes like this. The .pythonrc.py file is meant for interactive python sessions. A zopectl debug session sure feels like an interactive python session. In fact, it is an interactive python session, as zopectl debug uses the -i option to let python execute some statements and then drop into in interactive mode; but that is a special case, so your user file does not get loaded. But the general idea remains: it would be handy to load those user defaults. Of course that would only be handy in debug mode, not in other zopectl modes.

How do we do this? We change the $SOFTWARE_HOME/Zope2/Startup/zopectl.py file:

    def do_debug(self, arg):
        cmdline = self.get_startup_cmd(self.options.python + ' -i',
                                       'import user;' +
                                       'import Zope2; app=Zope2.app()')
        print ('Starting debugger (the name "app" is bound to the top-level '
               'Zope object)')
        os.system(cmdline)

The addition of import user is the only change there. For reference, the command line then becomes (split into a few lines here):

    /usr/bin/python2.3 -i -c "from Zope2 import configure;\
    configure(\'/home/maurits/instances/eXtremeManagement/etc/zope.conf\');\
    import user;import Zope2; app=Zope2.app()"

Since we do this in the do_debug function, this only gets used when you do a zopectl debug, not in a zopectl start or fg. So it should be safe, as far as I can tell.

Actually, after rereading I see that Jacob put something very similar into his above linked article.

So how about it? Can this go into Zope?

I can imagine that there are not too many people who know both conferences. At least I wore the T-shirt from EuroPython at New Wine and that only got blank looks. A short introduction seems in order then. Python is a programming language; I use it daily for my job. EuroPython is a yearly conference in Europe for python users and developers, sometimes affectionately called pythonistas. New Wine is a Christian movement that originated in England and has made the leap to the Netherlands and other countries. Their goal is: equipping churches to see Jesus' kingdom grow. Last week I visited the Dutch Summer Conference for the third year in a row.

As an aside: I haven't quite finished adding to this website summaries of all the talks at EuroPython that I went to. I plan to do that eventually, but first I want to migrate my home grown Zope 2 weblog to Plone plus Quills. If you don't know what I am talking about, then you are probably more in the New Wine camp than in the EuroPython camp. ;-) Anyway, you can expect more EuroPython stuff to appear here and also more New Wine summaries, although I will probably do those in Dutch only. I'll try to make it easier to view just the English weblog entries for those of you who are just as good in Dutch as the English speakers at the New Wine conference. :) Right, on to the comparison.

There are of course lots of similarities between the two conferences. You learn a lot, you meet new people, you get to know friends better, there is a good atmosphere and you have a lot of fun.

There are also marked differences:

• At EuroPython you get to know python better; at New Wine you get to know God better.
• At EuroPython no people got healed as far as I know; at New Wine several people got healed. I know at least of three people who got their foot healed; one person's back has improved considerably; one little girl who got bullied at school the whole year was now filled with joy.
• At EuroPython you learn to program the way Guido indented it (note for New Winers: that's not a spelling error); at New Wine you learn to live the way God intended it (note for pythonistas: that's also not a spelling error). I know I made some happy progress here, thanks to a good talk with Michelle and a seminar by Paulien Zeeman. All glory to God though.

By the way, Guido and God are much alike: they are both Benevolent Dictators For Life. For God life is a bit longer though. Oh, and they are both Dutch. ;)

If forced to choose, I would pick the New Wine Summer Conference any day. Except in winter, as that wouldn't make much sense. ;) Preferably though I would again visit them both next year.

Premature optimization is the root of all evil - C.A.R. Hoare

• Slow startup times are okay if you hardly ever start the program.
• Speed may not be important for a nightly cronjob.
• Actual user experience: does the program feel slow?

Plan:

1. Get the architecture right
2. if bugs: fix them
3. while code is too slow:
   • find worst bottleneck by profiling
   • try to optimize, running unit tests
   • if not faster: undo last code changes

Bottlenecks:

• maybe just your processor, memory, network, I/O, database problem
• Use OS tools, e.g. time, top, dstat, gkrellm, xosview to see in which program the problem is.
• python tools: profile (cprofile in python 2.5), hotshot, print statements

Big O notation

• differentiates between slower and faster algorithms, when the dataset imcreases.

Try to avoid O(n^2) and slower algorithms for large sizes of n.

Performance may be less important than code readability and maintainability.

Strategies:

• change algorithms
• change the architecture
• avoid nested loops
• move loop-invariant code out of the loop
• update only changed data
• cache instead of recompute or reload (but this can be error-prone)
• conversely, keep things out of the cache, as this might exhaust memory and start eating into disk space.
• Use multithreading for I/O bound code

File operations optimization:

• read file completely and then process if it is small
• read and process line by line if it is large
• use database instead of flat files

Python specific:

• Use python -O
• avoid exec and eval
• avoid from module import *
• shortcut namespace searches (e.g. opj = os.path.join)
• change code to use C-coded objects: lists, tuples, dictionaries, sets.

Don't waste developer time on unnecessary optimizations.