Start

You have a website that is still running fine on Plone 3.0.6. (The story is the same or similar for Plone 2.x, 3.0.x and 3.1.x.) You have a buildout.cfg like this:

[buildout]
parts = instance
versions = versions
find-links =
    http://dist.plone.org
    http://dist.plone.org/thirdparty
eggs =
    elementtree

[versions]
plone.recipe.plone = 3.0.6
plone.recipe.zope2instance = 3.6

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
fake-zope-eggs = true
url = ${plone:zope2-url}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:secret
http-address = 8080
eggs =
    ${buildout:eggs}
    ${plone:eggs}
zcml =
products = ${plone:products}

Error

Now you want to add an issue tracker product to this website. Okay, you add Products.Poi to the eggs:

[buildout]
...
eggs =
    elementtree
    Products.Poi

You run bin/buildout again and the result is an error:

Uninstalling instance.
Updating plone.
Updating zope2.
Updating fake eggs
Installing instance.
While:
  Installing instance.
Error: There is a version conflict.
We already have: plone.app.layout 1.0.5
but Plone 4.0b1 requires 'plone.app.layout>=1.1.7dev-r23744'.

This is not the result that you hope to get. Seeing this error you might think you need to do something about that plone.app.layout version. But the real error here is that bin/buildout tries to install Plone version 4, where the buildout.cfg has Plone version 3.0.6. It does this, because the added Products.Poi package specifies a dependency on Plone 4, specifically Plone>=4.0a1.

So why does bin/buildout not quit with a more informative error, like:

Error: There is a version conflict. We already have: Plone 3.0.6 but Products.Poi requires 'Plone>=4.0a1'.

The thing is, bin/buildout has no way of knowing that you have version 3.0.6, because this Plone version is not "eggified". Simply speaking, what this means is that there is no link to Plone 3.0.6 on this page: http://pypi.python.org/simple/Plone. There you will only find Plone 3.2 and higher, as that is the first eggified Plone version. For Plone 3.0.x and 3.1.x, the plone.recipe.plone buildout recipe is used to pull Plone and the other not-eggified products into your buildout. But this does not help when a third party package has a dependency on Plone: bin/buildout will see this and pull in the latest Plone egg it can find, resulting in errors like above.

Fake eggs

So how do we solve this? We could try to fake having a Plone egg. plone.recipe.zope2install already fakes a Zope2 egg, and lots of zope.*, zope.app.*, and other eggs, next to its main task of actually installing Zope2. You could say that plone.recipe.plone is a workaround for the not-eggified Plone and ``plone.recipe.zope2install is a workaround for the not-eggified Zope2. So, how could you fake a Plone egg? You add Plone to the additional-fake-eggs option of the recipe:

[zope2]
recipe = plone.recipe.zope2install
fake-zope-eggs = true
url = ${plone:zope2-url}
additional-fake-eggs =
    Plone

If you comment out the Products.Poi egg in buildout.cfg for a moment and rerun bin/buildout you will now have a file fake-eggs/Plone/Plone.egg-info with these contents:

Metadata-Version: 1.0
Name: Plone
Version: 0.0

If you do not like that version string 0.0, you could specify a different version in the recipe config:

additional-fake-eggs =
    Plone = 3.0.6

But of course later you will decide to update your buildout.cfg to use plone.recipe.plone=3.1.7 and will forget to update the additional fake egg version, leading to confusion, so you might as will accept the default 0.0 version.

Anyway, we have a fake egg now, but does it actually help us? Errr... no. Make sure Products.Poi is listed in the eggs again and rerun buildout:

Error: There is a version conflict.
We already have: plone.app.layout 1.0.5
but Plone 4.0b1 requires 'plone.app.layout>=1.1.7dev-r23744'.

That is the same error we got earlier. So why did it not work? That is easy: the picked Products.Poi version (2.0a1 at the time of writing) depends on Plone>=4.0a1, like we mentioned before. Plone 0.0 is available in the buildout due to our additional fake egg, but this version does not meet the requirements of Products.Poi, so bin/buildout gets the latest available Plone version instead, leading to problems.

Our previous solution would have worked if the picked Products.Poi version would have specified a dependency on any Plone version (Plone) or for example any Plone version lower than 4 (Plone<=4.0dev). So it might have its merits, but it is not enough in our case. In fact, I can imagine that some add-on products specify a dependency on Plone expecting to get at least Plone version 3.2, as that is the first eggified Plone version. Having a fake Plone egg would go against that expectation and might cause problems. So if you use this, use it with care.

Pin Plone

Our problems is still not solved. But the solution is only one line away. We add a version pin for Plone:

[buildout]
...
versions = versions
...

[versions]
plone.recipe.plone = 3.0.6
plone.recipe.zope2instance = 3.6
Plone = 0.0

When we now run bin/buildout again with Products.Poi in the eggs, we get this error:

The version, 0.0, is not consistent with the requirement, 'Plone>=4.0a1'.
While:
  Installing instance.
Error: Bad version 0.0

This error message is still not ideal, but at least this does not lead us on a wild goose chase after plone.app.layout, but clearly points out that some package depends on a newer Plone version than we want.

Debugging

If you now run bin/buildout in verbose mode by adding one or more v options, you get more info, which hopefully helps in debugging this (bin/buildout -v):

Getting required 'collective.watcherlist>=0.2'
  required by Products.Poi 2.0a1.
We have the best distribution that satisfies 'collective.watcherlist>=0.2'.
Picked: collective.watcherlist = 0.2
Getting required 'collective.autopermission'
  required by Products.Poi 2.0a1.
We have the best distribution that satisfies 'collective.autopermission'.
Picked: collective.autopermission = 1.0b1
Getting required 'Products.DataGridField>=1.8a1'
  required by Products.Poi 2.0a1.
We have the best distribution that satisfies 'Products.DataGridField>=1.8a1'.
Picked: Products.DataGridField = 1.8a1
Getting required 'Products.AddRemoveWidget>=1.4.2'
  required by Products.Poi 2.0a1.
We have the best distribution that satisfies 'Products.AddRemoveWidget>=1.4.2'.
Picked: Products.AddRemoveWidget = 1.4.2
The version, 0.0, is not consistent with the requirement, 'Plone>=4.0a1'.
While:
  Installing instance.
Error: Bad version 0.0

At this point, without further information, I would probably work my way up from the bottom and start investigating if Products.AddRemoveWidget 1.4.2 specifies a dependency on Plone (which it does not), and then checking Products.Poi 2.0a1 (bingo). Of course in this case we do know that the buildout worked fine before and is throwing errors after adding Products.Poi to the eggs. So starting the investigation with Products.Poi would make the most sense here. But knowing where to start looking is a matter of knowledge, experience and luck. Also: do not change too many things at the same time: do not update to a new Plone and Zope version and at the same time add ten new products in your buildout.cfg; it will only confuse you when things start going wrong.

In this case, at http://pypi.python.org/pypi/Products.Poi you currently see a list pointing to versions 2.0a1 and 1.2.8. Version 1.2.8 does not specify a dependency on Plone (speaking as Poi maintainer: nor will future 1.2.x versions), so you pin that version and things should work again:

[versions]
plone.recipe.plone = 3.0.6
plone.recipe.zope2instance = 3.6
Plone = 0.0
Products.Poi = 1.2.8

Dump all picked versions

Indeed, running bin/buildout now works. This might change though when one of the other not-pinned packages is updated and has unwanted changes or starts depending on Plone 4. So to top this overly long example off, we will add the buildout.dumppickedversions buildout extension, run bin/buildout and copy the reported versions to our buildout.cfg, resulting in the following file:

[buildout]
extensions = buildout.dumppickedversions
parts = instance
versions = versions
find-links =
    http://dist.plone.org
    http://dist.plone.org/thirdparty
eggs =
    elementtree
    Products.Poi

[versions]
Plone = 0.0
Products.AddRemoveWidget = 1.4.2
Products.DataGridField = 1.6.1
Products.Poi = 1.2.8
buildout.dumppickedversions = 0.4
distribute = 0.6.10
elementtree = 1.2.7-20070827-preview
plone.recipe.distros = 1.5
plone.recipe.plone = 3.0.6
plone.recipe.zope2install = 3.2
plone.recipe.zope2instance = 3.6
python-openid = 2.0.1
zc.buildout = 1.4.3
zc.recipe.egg = 1.2.2

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
fake-zope-eggs = true
url = ${plone:zope2-url}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:secret
http-address = 8080
eggs =
    ${buildout:eggs}
    ${plone:eggs}
zcml =
products = ${plone:products}

Fake it after all?

Incidentally, if you have an add-on product that simply depends on Plone without a minimum version, you would now get an error like this when running bin/buildout:

Getting distribution for 'Plone==0.0'.
While:
  Installing instance.
  Getting distribution for 'Plone==0.0'.
Error: Couldn't find a distribution for 'Plone==0.0'.

If you understand what you are doing you can still use the fake egg trick from earlier to fix this. But probably you should instead figure out which add-on is pulling in Plone, and see if an earlier version of that add-on is available that works with your Plone version without having to resort to faking eggs.

Note that for proper fake eggs, you will want to use at least version 3.0 of plone.recipe.zope2install, and probably the latest 3.2.

Summary

To avoid new Plone versions from being pulled in by add-on products when you are on a not-eggified Plone version (2.x, 3.0.x, 3.1.x), just add a version pin for Plone:

[buildout]
...
versions = versions
...

[versions]
Plone = 0.0

This will not magically solve your buildout problems. Basically, this tells bin/buildout that it should quit with an error when a package is depending on a Plone version. Hopefully the lines before the error message have a hint about which package this is; otherwise try bin/buildout -v to get more verbose information.

Konrad Delong: Concurrency in Python

Student at the Vrije Universiteit Amsterdam, via the Erasmus exchange program; I come from Poland. I am writing my master thesis on concurrency in python. Concurrency is about simultaneous computations, potentially interacting with each other. Could be running on separate cores or cpus.

Python has two modules for this: thread (low level, in C), and threading (more high level, more advanced threading primitives, more object oriented).

A lot of people do not like threading in Python because of the Global Interpreter Lock (GIL). Only a single thread can run code at one point. Other threads can start any time. Every 100 ticks (by default) the interpreter lock is released so another thread can take over. Also done on IO blocking operations. What this means is that you have no gains on multicore systems. Performance can even get worse for multiple cores, because threads battle for the GIL. Also, the latency in response to events is larger because of the GIL. On single core systems the performance with the GIL is quite good. It is a safe environment for C extensions: you can make some handy assumptions about the environment. Also, you do not have to use threading.

Some people try to remove the GIL. Greg Stein tried it; slower for single core machines, hardly faster on multicore; similar for python-safethread. There is PyPy and Unladen Swallow. There is also a policy on dropping GIL in cPython, which can be used. Other solutions: processes instead of threads; Jython and IronPython, Async.

The multiprocessing module (python 2.6) looks a lot like threading. Almost the same, but not entirely. No implicit shared memory (global variables, imported modules). There needs to be some interprocess communication (IPC). Data needs to be serializable (picklable).

Asynchronous solutions have been present in python for some time, like the standard asyncore module. The Twisted web server does this explicitly. It has a reactor loop that registers callbacks that other code should call when it is ready with processing. Other option: greenlets; all threads run in the same interpreter, so the same thread. See modules Eventlet, Gevent, PyEvent; quite a lot happening there recently. Lower level: select, epoll, kqueue for more directliy talking to the operating system. Also look at Kamaelia, and Stackless python. These solutions can be better when you would otherwise need thousands of threads. Some of this is used by big online games, like Eve Online.

See the slides.

Albert Visser

From Apeldoorn. Started as programmer in Cobol. You need a jcl file (stream) to make an executable program. I made a program to create that jcl automatically.

Roald de Vries: Properties and methods

From Go2People. With @property you turn a method into a property. How does that work? It is not an object property, but a class property. But the class property is not the same as the object property, which is not what you would expect. Descriptors can optionally implement __set__ and __delete__. In the attribute search order, python does some checks to see what should be returned, and this is where the descriptors take their place. Functions are always descriptors. See among others Shalabh Chaturvedi: Python Attributes and Methods.

Roland van Laar: how to open beer bottles

This was a demo about opening beer bottles without an opener. Always useful information during a PUN. :-)

Gijs Molenaar: Computer Vision with Python

From http://gijs.pythonic.nl. I am studying still, and doing things with computer vision and python. Computer vision means: extracting information from images. An image is a matrix of pixels. Things you can do, are: convert the color space (Hue Saturation Intensity), so you can for example focus on just one colour; thresholding; clustering; line, geometry or edge detection. Usage can be: optical character recognition, human computer interfaces, surveillance, augmented reality, or even an artistic toy. In The Netherlands we have section control (trajectcontrole) on some roads now: check if a car keeps to the maximum speed over a certain section of road (basically turning it into maximum average speed). More info: book Computer Vision: Algorithms and Applications, by Richard Szeliski from Microsoft Research; see http://research.microsoft.com/en-us/um/people/szeliski/Book/

About OpenCV: Open Computer Vision. It is open source, over 500 functions, originally developed by Intel, for Windows, Linux and Mac. Structure: HighGUI module for the IO and the GUI, make windows, sliders, webcam interface; CXcore module, for datatypes and basic functions; CV module for all the interesting computer vision functions; ML for Machine Learning, for analyzing data that comes out of the CV.

He demoed a program, of just a screen full of code, that captured live images from his webcam and did line recognition.

Three APIs: new python wrapper (native C++), old python wrapper (using swig), PyOpenCV (Boost/Bjam + numpy). The old wrapper is usually available on your OS as a package (aptitude install python-opencv, port install opencv). I have packages for the new wrapper ready for upload. Manual compilation: advantage: full control, some optimizations. Book: Learning OpenCV.

Problems: a lot of people report problems with installations; sometimes buggy, with segmentation faults, in which case you usually need to use the code a bit differently; sometimes undocumented; almost useless mailing lists, far more questions than answers, except when you have a really interesting question.

Presentations and examples can be downloaded.

Klaas van Schelven: virtualenv and pip

A virtualenvironment (virtualenv) is an isolated environment where you can install packages with pip, without bothering already installed packages in other environment. So for instance you can use this as an easy sandbox for safe testing of an unknown package. virtualenv --no-site-packages dirname creates a virtualenv in the directory dirname. cd dirname; source bin/activate. Set export PIP_RESPECT_VIRTUALENV=true somewhere in your ~/.bashrc. This way pip install mypackage will figure out if you are in a virtualenv and act accordingly. With pip freeze > production_ready.txt you can get the currently used package list with the correct versions, which you can reuse later, with pip install -r production_ready.txt.

Remark from the crowd: never remove old versions of packages from pypi or from wherever; it breaks peoples software. So please please never do this.

Guido Wesdorp: pydirs

From http://pragmagik.com. pydirs is a simple python object database. First ideas: 'svnodb', an svn object database, but subversion turned out to be a nuisance. So now pydirs. Simple code base, less than 500 lines. Simple explorable storage format with just directories and basic datatypes as text files. Great for debugging as you can use the command line tools. Everything else is stored as pickles. API: typeregistry, database object, session object (get, commit, rollback), pydiritem object. Currently quite stable, and used in production. I hope to release it very soon.

Update: it has been released here: http://johnnydebris.net/pydirs.txt

See my brother Reinout's weblog for another summary.

Chapter 1: Creating Courses

This is about creative application of the default Plone content types. It explains how to use some features of standard Plone, like collections, large folders and default pages. Recommended for Esther (Partner), Robin (Marketing/Sales), and Angela (Management Assistant).

Chapter 2: Calendaring

The Plone4ArtistsCalendar product is used as a nice way of showing events; this looks like a good solution for sites with lots of events. The chapter teaches best practices for using events and collections. You also get some tips about how best to apply security and permissions. Recommended for Esther (Partner), Robin (Marketing/Sales), and perhaps Mirella (Project Manager), mostly because they may be doing content management on some client sites or on our own site.

Chapter 3: Showcasing Personnel with Faculty/Staff Directory

The add-on product Faculty/Staff Directory can be seen as a departmental website in a box. It uses membrane, remember and relations. It is a versatile product, which could be useful in several occasions. Recommended for Jean-Paul (Partner) and Robin (Marketing/Sales): they should know this product is available; also recommended for the programmers (Mark, Fred, Vincent, me): they should see if this product fits the bill, before making something from scratch. It can also serve as a good example of how to use the membrane and remember products.

Chapter 4: Extending Faculty/Staff Directory

The standard product shown in the previous chapter is now extended using archetypes.schemaextender. Recommended for Mark, Fred and Vincent (programmers): look at this to freshen up your knowledge of archetypes.schemaextender (or for your first look at this).

Chapter 5: Blogs and Forums

This is a look at the out-of-the-box blogging potential of Plone (which may be enough), and at add-on products. Erik calls these "free as in puppies": they are cute, but you should be aware that they come with responsibilities. All add-on products that you may want to add in your site, should first be tested on a copy of that site, including trying to uninstall it. The QuillsEnabled and Scrawl products are introduced for blogging, including using them in combination if needed. For forums the judgement is clear: use Ploneboard. Recommended for Mark as consultant and Fred as consultant/trainer.

Chapter 6: Embedding Audio and Video

For embedding audio and video, Erik recommends collective.flowplayer as the most flexible and trouble-free solution. Plone4ArtistsVideo is mentioned once for podcasting support, but perhaps Plone4ArtistsAudio is meant, as that one is explained. In some cases, for example embedding YouTube videos, you need to write some pure html yourself, so enabling the object and embed tags is explained; I did not notice before that there was an embed-tab button that you can enable in Site Setup, Visual Editor, Toolbar. Again recommended for Mark as consultant and Fred as trainer.

Chapter 7: Creating Forms Fast

I already wrote about this chapter in my first impressions. Recommended for Esther (Partner), Robin (Marketing/Sales), Mark (Consultant), Fred (Trainer). This was a handy chapter for me as well; I just know Mirella has ten times more experience here than I do. :-)

Chapter 8: Styling Your Site

This chapter explains the Zope 2 way versus the Zope 3 way of theming. Depending on which parts of your site you want to style, you will need to learn both ways. You will learn about through-the-web versus development on the file system. You use paster for creating a basic theme product. Erik shows z3c.jbot as the easiest way of overriding templates from Plone's sometimes difficult viewlet and portlet machinery. Recommended for Mirella and Laurens (CSS Specialists) for perhaps a better understanding of why some things need to be done in a certain way.

Chapter 9: Going Live

This is all about buildout, a zeo setup, CacheFu, Squid and Apache. Recommended for the programmers when dealing with Squid instead of the varnish that we mostly use; and a fresh look at the CacheFu settings would not hurt.

Chapter 10: Maintenance, Backups, and Upgrades

The instructions to pack manually give wrong directions, which will have you end up in the ZMI in the Plone Site root, without a Zope Control Panel in sight. Instead you should simply go to Site Setup and then to Maintenance. Other than that, this chapter has no surprises, but if you come working for us as programming sysadmin this is the first chapter you should read.

Conclusion

Plone 3 for Education is a very practical book. It shows you a safe route through Plone land, giving solid advice that will keep you out of the pitfalls of this great CMS.

Disclaimer: I got this book for free from Packt Publishing in exchange for a review. Ordering the book via one of the links in this article will land me some money.