Creating repeatable buildouts
When you create a buildout for a production website, you want to be able to recreate that exact buildout one year from now on a different server. How do you do that? Earlier version, kept for historical reasons
I made a better, easier version of this weblog entry. Please read that instead, unless you are interested in some strange buildout errors that can occur when you are doing things a bit too difficult, like I am doing here.
When you create a buildout for a production website, you want to be able to recreate that exact buildout one year from now on a different server. How do you do that?
Background
I assume you know what buildout is, otherwise you would probably not be reading this. Martin Aspeli has written a tutorial on Managing projects with zc.buildout. This should be your first stop for learning more about buildout, at least if you are a Plone developer.
My brother and colleague Reinout has written a weblog entry about the Buildout development/production strategy that we are starting to use at Zest Software. The goal is to use the same buildout for both development and production deployment.
As the buildout should be repeatable, you may want to read the repeatable test file from the zc.buildout package.
Definitions
Now I am going to write about creating a really stable and repeatable buildout. In Reinout's story that would fit in with creating a stable.cfg. If you are not concerning yourself with development or if you have a different buildout for production deployment, this would just be the buildout.cfg file.
Let's begin with some perhaps arbitrary definitions.
With stable I mean: you run the buildout script in a directory; one year later you run buildout for the second time; you get the same result as one year earlier. Use case: by accident you removed the entire parts directory and you need to rebuild it.
With repeatable I mean: you run the buildout script in a directory; one year later you run the buildout on a completely different server; both directories are exactly the same. Use case: you want to move the zope instance of a customer to a new server.
Creating the buildout
What I write in this entry should be applicable to any buildout with or without Plone. My main focus and example is a Plone buildout though. You can fix up a current buildout of course, but here let's start with a clean default buildout for Plone 3, which you can create with paster:
paster create -t plone3_buildout test cd test python2.4 bootstrap.py
No newer packages
When running bin/buildout by default it picks the newest package it can find for any dependency. We can tell buildout to first look if some version of that package is already available:
[buildout] newest = false
This way, a second call of bin/buildout will not try to get any new packages. So it will finish quickly and just keep the current packages. This already makes the buildout stable in the sense that a new run of bin/buildout in that same directory will give you no new packages.
But if you want to recreate that buildout in a different directory or even a different server, you can get other versions. For starters, currently this buildout config will give you Plone 3.0.5, but a few months from now you will probably get Plone 3.1. So this is not yet a repeatable buildout.
Simple index
As an aside, while we are in the buildout section, let's quickly specify a simpler (read: faster) index than the default CheeseShop:
[buildout] newest = false index = http://download.zope.org/ppix
See my brother's weblog entry on ppix instead of pypi from now on.
By the way, this has nothing to do with making your buildout more stable or repeatable.
Pinning the plone version
Let's go get some more stability in our buildout. First things first: we pick a Plone version. Change this:
recipe = plone.recipe.plone
into this:
recipe = plone.recipe.plone == 3.0.5
This recipe has strict versions for the Plone products and packages that make up a Plone release, so for example Products.CMFPlone, Products.CMFCore, the various plone.* packages, etcetera.
And this has a zope2-url property that tells the plone.recipe.zope2install recipe that it should use Zope 2.10.5.
So the most important versions have been pinned now.
Pinning extra products
You can of course also pin any extra products that you want to use in your buildout. Best is to use an official release, such as a tar ball:
[productdistros]
recipe = plone.recipe.distros
urls =
http://plone.org/products/poi/releases/1.1/poi_1.1.tgz
or a checkout of a subversion tag:
[productcheckouts] recipe = infrae.subversion urls = http://svn.plone.org/svn/collective/eXtremeManagement/tags/1.5.2/ eXtremeManagement
If for some reason there is no tag you can use, you can still specify a revision in the url with the @ sign. Thanks to Guido Wesdorp for pointing this out:
[productcheckouts]
recipe = infrae.subversion
urls =
http://getpaid.googlecode.com/svn/trunk/products/PloneGetPaid@1132 PloneGetPaid
You can of course also pin the eggs of packages that you need, which we will see below. For now let's keep the example simple here and remove these products again from our buildout.cfg.
Knowing which versions are not pinned
Wichert Akkerman came up with the following one-liner to get a list of versions that are not pinned by you, but picked by buildout:
bin/buildout -Novvvvv |sed -ne 's/^Picked: //p' | sort | uniq
Currently this returns this list:
elementtree = 1.2.7-20070827-preview plone.recipe.distros = 1.3 plone.recipe.zope2install = 1.2 plone.recipe.zope2instance = 1.3 python-openid = 2.0.1 setuptools = 0.6c7 zc.recipe.egg = 1.0.0
This basically means that the mentioned packages are not pinned by our buildout config, but chosen (picked) by buildout. This means that they are not stable yet: rerunning this script in a few months time will likely give you different versions.
Pinning a package
We start easily enough by pin elementtree. Instead of this:
eggs =
elementtree
we write this:
eggs =
elementtree == 1.2.7-20070827-preview
As an aside, note that the top directive eggs has one equals sign after it and the option elementtree below it has a double equals sign. The point is that in the options you write a test, so you could also choose greater than (>=) or smaller than (<=). But we want stability, so we keep the double equals sign.
If you now run bin/buildout and then run that one-liner from Wichert again you will now see that the elementtree line has disappeared from the output: buildout no longer picks that version for us as we have pinned it. We are getting closer to stability!
Timeout
At this point it is good to say that you could stop here: all Zope and Plone products and packages are pinned, which is the most important. Does it really matter if one year from now plone.recipe.zope2instance is at version 2.0 instead of 1.3? Probably not. But there is still room for more stability and repeatability. So if you are still interested, let's continue our quest.
Pinning recipes
Now we start pinning the other recipes that are in the output. Find these four lines in your buildout.cfg (they will not be right below each other):
recipe = plone.recipe.zope2install recipe = plone.recipe.distros recipe = plone.recipe.zope2instance recipe = zc.recipe.egg
and pin them to specific versions:
recipe = plone.recipe.zope2install == 1.2 recipe = plone.recipe.distros == 1.3 recipe = plone.recipe.zope2instance == 1.3 recipe = zc.recipe.egg == 1.0.0
Now we run the one-liner again and get:
plone.recipe.distros = 1.3 python-openid = 2.0.1 setuptools = 0.6c7 zc.recipe.egg = 1.0.0
This may not be what you had expected. We have pinned plone.recipe.distros and zc.recipe.egg right? That is correct, but look at this partial output of bin/buildout with the verbose option:
$ bin/buildout -v Installing 'plone.recipe.plone == 3.0.5'. We have the distribution that satisfies 'plone.recipe.plone==3.0.5'. Getting required 'plone.recipe.distros' required by plone.recipe.plone 3.0.5. Picked: plone.recipe.distros = 1.3 Getting required 'zc.recipe.egg' required by plone.recipe.plone 3.0.5. Picked: zc.recipe.egg = 1.0.0
What happens here is that plone.recipe.plone depends on two other recipes. They are in the install_requires option of the setup.py file of the recipe. So buildout itself picks a version of those dependencies. It does not look at our pinnings at it is handling the [plone] part of buildout.cfg now and not the [productdistros] or [zopepy] parts that try to pin versions for those recipes.
In fact, you can get a conflict with this. Temporarily we try to pin plone.recipe.distros to an earlier version:
[productdistros] recipe = plone.recipe.distros == 0.3
We run bin/buildout:
While:
Installing.
Getting section productdistros.
Initializing section productdistros.
Loading recipe 'plone.recipe.distros == 0.3'.
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
VersionConflict:
(plone.recipe.distros 1.3
(.../plone.recipe.distros-1.3-py2.4.egg),
Requirement.parse('plone.recipe.distros==0.3'))
In this case buildout already picked version 1.3 and now we tell it that we require 0.3 so this conflicts. How do we solve that? We change the order in which the buildout parts are executed. Currently it is this:
[buildout]
parts =
plone
zope2
productdistros
instance
zopepy
Now we make sure that productdistros and zopepy are above plone:
[buildout]
parts =
productdistros
zopepy
plone
zope2
instance
We run bin/buildout -v again:
Uninstalling productdistros.
While:
Installing.
Uninstalling productdistros.
Loading recipe 'plone.recipe.distros == 1.3'.
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
VersionConflict:
(plone.recipe.distros 0.3
(.../plone.recipe.distros-0.3-py2.4.egg),
Requirement.parse('plone.recipe.distros==1.3'))
Oops. Here we are uninstalling version 1.3 as that was installed by the previous buildout run and at the same time we want to install 0.3. buildout does not like this. Only solution I know: remove the .installed.cfg file that is automatically created by buildout. The function of that file is to keep track of what buildout has previously installed, so it knows if it should do any uninstalling or reinstalling or if it can just do nothing and finish within one second. In other words: removing that file should be okay. Anyway, you are keeping backups, right?
Five minutes later...
Right, we are keeping backups. We remove that .installed.cfg file and run bin/buildout again. This will take a bit longer now, as among others it compiles Zope again. At least it completes without conflicts now, and plone.recipe.distros is not picked by buildout anymore, but pinned by us.
For some reason zc.recipe.egg is still picked though. Ah, the [zopepy] part that pins this recipe contains this line:
extra-paths = ${zope2:location}/lib/python
So this part depends on the [zope2] part, which in turn depends on the [plone] part, which in turn depends on the zc.recipe.egg package, which is therefore picked by buildout, ignoring our pinning.
Wonderful.
Okay, seems like we have to resort to trickery. We introduce a new section [dummy-for-pinning]:
[dummy-for-pinning] recipe = zc.recipe.egg == 1.0.0 eggs = zc.recipe.egg == 1.0.0
Yes, I tried this and you need the version number in both lines. Now we simply put that section at the top of the buildout parts. And we move the [zopepy] section down, as it depends on those other parts anyway. So our parts now looks like this:
parts =
dummy-for-pinning
productdistros
plone
zope2
instance
zopepy
Pinning the rest
What does the one-liner tell us?:
$ bin/buildout -Novvvvv |sed -ne 's/^Picked: //p' | sort | uniq python-openid = 2.0.1 setuptools = 0.6c7 zc.buildout = 1.0.0
zc.buildout is a new one. This is a requirement of zc.recipe.egg, though I could not say why it did not end up in that list earlier. We can add all three remaining picked packages to our [dummy-for-pinning] section so it now looks like this:
[dummy-for-pinning]
recipe = zc.recipe.egg == 1.0.0
eggs =
zc.recipe.egg == 1.0.0
zc.buildout == 1.0.0
setuptools == 0.6c7
python-openid == 2.0.1
After this we hit the end of the road. zc.buildout is pinned, but setuptools and python-openid are still picked for us by buildout. The last possibility I can think of is adding those two to a [versions] section, like this:
[versions] setuptools = 0.6c7 python-openid = 2.0.1
But this has no noticeable effect. In fact, when I change the setuptools version to 0.6c6 (so 6 instead of 7) in both cases, I get this output from buildout:
Installing dummy-for-pinning. Installing 'zc.recipe.egg == 1.0.0', 'zc.buildout == 1.0.0', 'setuptools == 0.6c6', 'python-openid == 2.0.1'. We have the distribution that satisfies 'zc.recipe.egg==1.0.0'. We have the distribution that satisfies 'zc.buildout==1.0.0'. We have the distribution that satisfies 'setuptools==0.6c6'. We have the distribution that satisfies 'python-openid==2.0.1'. ... Getting required 'setuptools' required by five.customerize 0.2. ... required by plone.app.kss 1.2.5. Picked: setuptools = 0.6c7
So our pinning of setuptools is ignored. The same is true for python-openid:
Getting required 'python-openid>=2.0.0,<2.0.999' required by plone.openid 1.0.1. Picked: python-openid = 2.0.1
One last resort: introduce a dummy package that requires a specific version of those two packages in its setup.py. We add it to our buildout and see that it gets used correctly:
We have the distribution that satisfies 'zest.recipe.dummy==0.3'. Getting required 'python-openid==2.0.1' required by zest.recipe.dummy 0.3. We have the distribution that satisfies 'python-openid==2.0.1'. Getting required 'setuptools==0.6c6' required by zest.recipe.dummy 0.3. We have the distribution that satisfies 'setuptools==0.6c6'.
But further on in the buildout nothing has changed and buildout still picks its own versions.
One final twist though: I removed both versions of setuptools from the eggs directory. I then ran buildout again. It again fetched version 6 and then claimed to have picked version 7, but the only package that I could actually find, was version 6. And the second time I ran buildout, it already had version 6, so it no longer picked version 7, but version 6.
Are you still with me? :-) Then maybe just a few more random notes before we get to the conclusions.
- Try this: create a new buildout with paster. Do the usual python2.4 bootstrap.py. You will now have downloaded version 7 of setuptools and version 1.0.0 of zc.buildout. The task of persuading the bootstrap to give you earlier (or later) versions of those two packages is left as an exercise to the reader. Assignments are due next week.
- What I had not realized yet: what you put in the [versions] section influences what goes in the bin/buildout script itself. I have not looked at what the end-effect here is.
- After some of these changes I have seen bin/buildout finish succesfully and then found that the bin/buildout file was gone. Running python2.4 bootstrap.py fixes that of course, but it is strange.
Conclusions
A buildout can be made very stable: just put newest = false in your buildout section. If one year later you accidentally remove the parts directory, you can rerun buildout and get your original directory back. If you remove your eggs though, you are in trouble.
Making a buildout repeatable is more difficult. The first easy step is pinning the plone.recipe.plone package. Any extra packages or products can be pinned quite easily as well.
What is difficult is pinning dependencies. If you do not do this correctly, your pinning will cause a conflict. The order in which the buildout parts or sections get executed is important here.
You will have to resort to trickery to get the last few packages pinned down. And even then it does not seem possible to pin really everything. But we are very close and I think there is a good chance that the remaining issues will get solved in buildout, setuptools or easy_install.
Bonus
plone.recipe.plone rigourously depends on specific packages and products. So what do you do if you want to use a newer version of just one or two packages? For instance, I did some fixes for plone.locking which are still not in Plone 3.0.5. And for multilingual sites you really want to use a newer version with an important bug fix. You might at first think that this would be enough:
[plone]
recipe = plone.recipe.plone == 3.0.5
eggs =
plone.locking == 1.0.5
plone.app.i18n == 1.0.2
But running bin/buildout then gives an error:
ValueError:
('Missing distribution spec', '==')
This is how you accomplish that:
[versions]
plone.locking = 1.0.5
plone.app.i18n = 1.0.2
[plone]
recipe = plone.recipe.plone
eggs =
plone.locking
plone.app.i18n
To start at the bottom: these lines basically erase the version pinning of those two packages that is in the recipe. After that, the pinning in the [versions] section can take effect.
But in fact, nothing happens just yet. This is because we are running buildout in non-newest mode (newest = false in the [buildout] section). But here we actually do not want stability: we want newer versions! So we run buildout in the newest mode, either by temporarily setting newest = true or by calling bin/buildout -n once. Now we get the new packages that we want. Since we have pinned almost everything else, this should be quite safe.
The final version
Let's end with showing what our buildout.cfg now looks like. Note here that the mention of setuptools, zc.buildout and python-openid in [versions] or [dummy-for-pinning] may not be too useful. And I will not fault you if you skip that [dummy-for-pinning] section entirely. The reported picked versions are now:
plone.app.i18n = 1.0.2 plone.locking = 1.0.5 python-openid = 2.0.1 setuptools = 0.6c7
And here is the file itself:
[buildout]
newest = false
index = http://download.zope.org/ppix
parts =
dummy-for-pinning
productdistros
plone
zope2
instance
zopepy
find-links =
http://dist.plone.org
http://download.zope.org/ppix/
http://download.zope.org/distribution/
http://effbot.org/downloads
# Add additional eggs here
# elementtree is required by Plone
eggs =
elementtree == 1.2.7-20070827-preview
[dummy-for-pinning]
recipe = zc.recipe.egg == 1.0.0
eggs =
python-openid == 2.0.1
setuptools == 0.6c7
zc.buildout == 1.0.0
zc.recipe.egg == 1.0.0
[versions]
setuptools = 0.6c7
zc.buildout = 1.0.0
plone.locking = 1.0.5
plone.app.i18n = 1.0.2
[plone]
recipe = plone.recipe.plone == 3.0.5
eggs =
plone.locking
plone.app.i18n
[zope2]
recipe = plone.recipe.zope2install == 1.2
url = ${plone:zope2-url}
[productdistros]
recipe = plone.recipe.distros == 1.3
urls =
nested-packages =
version-suffix-packages =
[instance]
recipe = plone.recipe.zope2instance == 1.3
zope2-location = ${zope2:location}
user = admin:
http-address = 8080
#debug-mode = on
#verbose-security = on
# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
${buildout:eggs}
${plone:eggs}
# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =
products =
${buildout:directory}/products
${productdistros:location}
${plone:products}
[zopepy]
recipe = zc.recipe.egg == 1.0.0
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy