Version pinnings and fake eggs and allow-picked-versions

I posted this to the Plone product-developers list a few days ago, but got no reactions yet, so I thought I would give it a wider audience.

Main point of this post is to say: Hey, be aware that even if you think you have everything pinned in your buildout config, you might still be using unpinned eggs. And using unpinned eggs means that your buildout that runs fine today might give problems half a year later.

This is similar to a problem in the plonenext buildout and the plone.recipe.zope2instance 3.8 package discussed on the Plone core developers list this week. (3.9 has been released, solving that problem).

Observe the following small buildout.cfg. It does not do anything useful, except that it showcases a possible problem:

[buildout]
extensions = buildout-versions
parts = zope2 test
versions = versions
# Next line is important!
allow-picked-versions = false

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.10.11/Zope-2.10.11-final.tgz
fake-zope-eggs = true

[test]
recipe = zc.recipe.testrunner
# Random egg as example
eggs = pep8

[versions]
buildout-versions = 1.2
distribute = 0.6.14
plone.recipe.zope2install = 3.2
zc.buildout = 1.4.3
pep8 = 0.5.0
zc.recipe.egg = 1.2.2
zc.recipe.testrunner = 1.3.0
zope.testrunner = 4.0.0b5

Put that in a directory with a bootstrap.py and start the buildout process:

$ python2.4 bootstrap.py
....
$ bin/buildout
Upgraded:
   zc.buildout version 1.4.3,
   distribute version 0.6.14;
restarting.
Generated script '/Users/mauritsvanrees/tmp/pin/bin/buildout'.
While:
   Installing.
   Getting section test.
   Initializing section test.
   Installing recipe zc.recipe.testrunner.
   Getting distribution for 'zope.interface'.
Error: Picked: zope.interface = 3.6.1

zc.recipe.testrunner depends on zope.testrunner, which depends on zope.interface and zope.exceptions. Since these packages have no version pin in the buildout and we have specified allow-picked-versions = false, buildout quits with an error.

But, those two dependencies are in the Zope2 tarball and will be available as fake eggs. Can't buildout use those fake eggs?

Yes, it can, if we set allow-picked-versions = true (or comment that line out). Rerun buildout and it works:

$ bin/buildout
Installing zope2.
Creating fake eggs
Installing test.
Generated script '/Users/mauritsvanrees/tmp/pin/bin/test'.
Versions had to be automatically picked.
The following part definition lists the versions picked:
[versions]

# Required by:
# zope.testrunner==4.0.0b5
zope.exceptions = 3.6.1

# Required by:
# zope.testrunner==4.0.0b5
zope.interface = 3.6.1

Those last lines are output by buildout-versions (variant of buildout.dumppickedversions). It correctly reports that the two mentioned eggs have been picked.

But... they are not used in the generated bin-script:

$ grep zope.interface bin/*
bin/test:  '/Users/mauritsvanrees/tmp/pin/fake-eggs/zope.interface',
$ grep zope.exceptions bin/*
bin/test:  '/Users/mauritsvanrees/tmp/pin/fake-eggs/zope.exceptions',

Now we could add the suggested versions to the [versions] section, set allow-picked-eggs = false again, and rerun the buildout. Result:

$ grep zope.interface bin/*
bin/test:  '.../zope.interface-3.6.1-py2.4-macosx-10.6-i386.egg',
$ grep zope.exceptions bin/*
bin/test:  '.../zope.exceptions-3.6.1-py2.4.egg',

OR: do not add those versions, but do set allow-picked-versions = false so you have the exact same buildout.cfg as in the beginning and this time bin/buildout 'magically' succeeds. It uses the fake eggs in the script and buildout-versions correctly does not report missing pins.

So: the same buildout.cfg crashes at first, and later succeeds. This is because the second time around the fake eggs are already there and the buildout process makes use of them.

What is the potential problem here? If you want to use the same buildout.cfg on a machine without an intermediate edit, you have some options:

1. Set allow-picked-versions = true. During the initialization phase of buildout you do use the latest zope.interface and zope.exceptions from pypi (which might have a bug, or for example might only work with python2.6) but at least they do not actually get used in any of the generated scripts. So you get the zope.interface and zope.exceptions from the Zope2 tarball, which is good.

2. Keep allow-picked-versions = false. Add the two version pins. Now you know exactly which eggs you use without being surprised by sudden incompatible releases. But in the resulting scripts you are not using the zope.interface and zope.exceptions form the original Zope2 tarball. So subtle things might go wrong. BTW, it might be better to use some older eggs instead.

3. Avoid using any recipes that have dependencies on fake Zope 2 eggs.

Neither of the solutions is ideal I would say. With Plone 4 we use Zope 2.12 which is fully 'eggified', so no fake eggs are needed anymore, so this problem goes away then.

For Plone 3 (or earlier) I think we have to live with this problem and everyone will have to choose one of the partial solutions and be smart enough to be able to handle possible future problems.

Or does someone see a better, real solution?

Reactions are welcome as always via the contact form or via email or in this case via the Plone product-developers list.

Update: Note that the order in which the buildout parts get installed does not matter here. Buildout first gets all the recipes for all the parts, and if one of those recipes depends on zope.interface, it already goes wrong at that point. So that is even before the __init__ method of the recipes is called, let alone the install method.

Update 2: If I first explicitly do 'bin/buildout install zope2' and then 'bin/buildout' then it actually works. I thought I had tried that already. Still not ideal I would say, but it's a good extra option to have. A buildout run with the original buildout.cfg from above could then succeed in two steps:

$ rm -rf develop-eggs/ fake-eggs/ .installed.cfg parts/
$ bin/buildout
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface'.
Error: Picked: zope.interface = 3.6.1
$ bin/buildout install zope2
Installing zope2.
Creating fake eggs
$ bin/buildout
Updating zope2.
Updating fake eggs
Installing test.
Generated script '/Users/mauritsvanrees/tmp/pin/bin/test'.

Update 3: With these rather old version pins (from 2007) there are no dependencies on zope.interface and zope.exceptions anymore. It might be fine to use those, or you might miss some newer features and bug fixes:

zc.recipe.testrunner = 1.0.0
zope.testing = 3.5.1 

Thanks to Florian Schulze for some questions and remarks that got me thinking a bit further.