Geir Bækholt talks at the Dutch Plone User Day 2010 in the Euromast, Rotterdam.

Plone 4 was released two weeks ago. It has taken longer than we thought, but we think it was worth the wait. It was meant to be a minor release. "Plone 5 is going to have large changes, so let's make this 4.0 release a little one." We totally failed at that. It is the biggest release ever.

To get new functionality into Plone, we use a 'PLIP' process. A PLIP is a PLone Improvement Proposal. The Framework Team accepts or rejects these PLIPs. For Plone 4 we had 58 plips, which is the biggest amount ever. The process was more documented this time, so that probably made it easier for people to get their improvement proposal in. In the end, 27 plips made it into Plone 4.0, some have been postponed.

The release manager of Plone 4 is Eric Steele; works at Penn State University in the USA. He probably had no idea what he stepped up for. ;-)

Lezing tijdens de Nederlandse Plone Gebruikersdag 2010 in de Euromast, met prachtig mistig uitzicht op het schitterende Rotterdam.

Van web '0.9' naar 2.0 voor http://www.exser.nl. Exser is een stichting opgericht door diverse partijen. Samenwerkingsprogramma's van overheid, wetenschap en bedrijven. In 2008 hadden we een statische website. Prima voor het eerste jaar dat we op de markt gingen, maar we waren toe aan iets nieuws. We wilden zelf de content aan kunnen passen, zonder dure technici in te hoeven schakelen. We wilden meer contact online faciliteren.

Onze wensen waren wat 'fuzzy', niet duidelijk gedefinieerd. We waren een nieuwe business en waren zelf nog van alles aan het leren. Vijf bureau's uitgenodigd. Vaak zaten in de offerte spelfouten en er werd niet in de klant ingeleefd, wat toch jammer is als onder aan de streep 35 tot 40 duizend euro staat. Lex van Sonderen van Proteon belde diverse keren op tijdens dit traject: "Zit ik zo in de goede richting?" Hij had dus wel aandacht voor de klant.

Uitdaging voor ons: goed leren omgaan met het CMS. Als projectleider stuurde ik op drie zaken: kwaliteit, kosten en snelheid, wat natuurlijk deels in tegenspraak is, dus aanleiding voor gesprekken met de websitebouwer. Daar zaten ook best pittige discussies bij, vooral over de kosten. Maar we konden open communiceren, dus dat leverde uiteindelijk geen probleem op. Uiteindelijk leverde het een mooie website op, met ons als tevreden klant.

Op de website staat onder andere een nieuwsbrief, linked-in integratie, twitter portlet, RSS feed. Waarschijnlijk technisch niet eens zo moeilijk, maar voor ons handig.

Tip voor ontwikkelaars: kruip in de huid van de klant. Deel het project op in goed gedefinieerde brokken functionaliteit. Lever dat op per iteratie. Laat regelmatig voortgang zien. Geef goed inzicht in de kosten en zeg het tijdig wanneer kosten boven het geplande budget dreigen uit te komen. De klant wil eigenlijk niet per uur betalen, maar wil een product krijgen voor een vaste prijs. Betalen voor output in plaats van uren. Denk daar over na. Ik verwacht dat over een aantal jaar de klant niet meer akkoord gaat met onderstaand antwoord: "Bij ons betaalt u voor uren, niet voor een product."

Stel jezelf vragen. Wie is onze klant? Waar ligt hij wakker van? Wat bieden we aan en welke behoefte vervullen we daarmee? Hoe gaat de klant daarvoor afrekenen? Hoe produceer je dat, wat is daarvoor nodig?

In de advocatuur beweren sommigen dat binnen een aantal jaar uurtje-factuurtje niet meer bestaat. Bedenk dus andere manieren van betaling. Laat uw strategische klanten meedenken over mogelijke verdienmodellen. Ik heb Lex ook aangeboden daar over mee te denken. Je kan hoge inschattingen geven, zodat je minder snel over de inschatting heen gaat, maar daar zit een grens aan, daar zal een stukje marktwerking in zitten. Kijk eens naar andere bedrijfstakken voor hun betaalmodellen.

Meer weten en leren? Samen met Nyenrode geeft Exser een masterclass Innoveren van diensten; start januari 2011.

Idee van Lex: niet een vaste prijs voor het hele project, maar per iteratie.

Opmerking Edith: wij zijn juist blij met uurtje-factuurtje. Heeft zeker z'n problemen. Maar bij betalen voor een product hadden we te weinig controle. Nu hebben we met Zest Software gewoon regelmatig contact en daarbij werkt uurtje-factuurtje nu prima.

Zie de slides.

I went for one day to the Living Statues Sprint in Arnhem, The Netherlands, hosted by Four Digits. I saw lots of new things:

• plone.app.collections: much more lightweight way to do Collections in Plone, probably included by default in Plone 4.1.
• plone.app.contentlisting: small package that should make it easier to have a listing of either brains or content items, making the standard Plone templates that do these listings much smaller.
• plone.app.standardtiles: package that implements lots of small bits and pieces of content that are already in Plone, but now in a more general way. They partly are like portlets, partly viewlets. These tiles (for example a content listing tile, which I worked on with Ralph) can be added to the layout of dexterity content types. It is a really flexible way of adding small pieces of html to your page. It is experimental for Plone 4; probably this will be how Plone 5 will work, using the Deco grid system.

Nice to have a look at the future of Plone. But first Plone 4.0, which should be out real soon now.

I posted this to the Plone product-developers list a few days ago, but got no reactions yet, so I thought I would give it a wider audience.

Main point of this post is to say: Hey, be aware that even if you think you have everything pinned in your buildout config, you might still be using unpinned eggs. And using unpinned eggs means that your buildout that runs fine today might give problems half a year later.

This is similar to a problem in the plonenext buildout and the plone.recipe.zope2instance 3.8 package discussed on the Plone core developers list this week. (3.9 has been released, solving that problem).

Observe the following small buildout.cfg. It does not do anything useful, except that it showcases a possible problem:

[buildout]
extensions = buildout-versions
parts = zope2 test
versions = versions
# Next line is important!
allow-picked-versions = false

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.10.11/Zope-2.10.11-final.tgz
fake-zope-eggs = true

[test]
recipe = zc.recipe.testrunner
# Random egg as example
eggs = pep8

[versions]
buildout-versions = 1.2
distribute = 0.6.14
plone.recipe.zope2install = 3.2
zc.buildout = 1.4.3
pep8 = 0.5.0
zc.recipe.egg = 1.2.2
zc.recipe.testrunner = 1.3.0
zope.testrunner = 4.0.0b5

Put that in a directory with a bootstrap.py and start the buildout process:

$ python2.4 bootstrap.py
....
$ bin/buildout
Upgraded:
   zc.buildout version 1.4.3,
   distribute version 0.6.14;
restarting.
Generated script '/Users/mauritsvanrees/tmp/pin/bin/buildout'.
While:
   Installing.
   Getting section test.
   Initializing section test.
   Installing recipe zc.recipe.testrunner.
   Getting distribution for 'zope.interface'.
Error: Picked: zope.interface = 3.6.1

zc.recipe.testrunner depends on zope.testrunner, which depends on zope.interface and zope.exceptions. Since these packages have no version pin in the buildout and we have specified allow-picked-versions = false, buildout quits with an error.

But, those two dependencies are in the Zope2 tarball and will be available as fake eggs. Can't buildout use those fake eggs?

Yes, it can, if we set allow-picked-versions = true (or comment that line out). Rerun buildout and it works:

$ bin/buildout
Installing zope2.
Creating fake eggs
Installing test.
Generated script '/Users/mauritsvanrees/tmp/pin/bin/test'.
Versions had to be automatically picked.
The following part definition lists the versions picked:
[versions]

# Required by:
# zope.testrunner==4.0.0b5
zope.exceptions = 3.6.1

# Required by:
# zope.testrunner==4.0.0b5
zope.interface = 3.6.1

Those last lines are output by buildout-versions (variant of buildout.dumppickedversions). It correctly reports that the two mentioned eggs have been picked.

But... they are not used in the generated bin-script:

$ grep zope.interface bin/*
bin/test:  '/Users/mauritsvanrees/tmp/pin/fake-eggs/zope.interface',
$ grep zope.exceptions bin/*
bin/test:  '/Users/mauritsvanrees/tmp/pin/fake-eggs/zope.exceptions',

Now we could add the suggested versions to the [versions] section, set allow-picked-eggs = false again, and rerun the buildout. Result:

$ grep zope.interface bin/*
bin/test:  '.../zope.interface-3.6.1-py2.4-macosx-10.6-i386.egg',
$ grep zope.exceptions bin/*
bin/test:  '.../zope.exceptions-3.6.1-py2.4.egg',

OR: do not add those versions, but do set allow-picked-versions = false so you have the exact same buildout.cfg as in the beginning and this time bin/buildout 'magically' succeeds. It uses the fake eggs in the script and buildout-versions correctly does not report missing pins.

So: the same buildout.cfg crashes at first, and later succeeds. This is because the second time around the fake eggs are already there and the buildout process makes use of them.

What is the potential problem here? If you want to use the same buildout.cfg on a machine without an intermediate edit, you have some options:

1. Set allow-picked-versions = true. During the initialization phase of buildout you do use the latest zope.interface and zope.exceptions from pypi (which might have a bug, or for example might only work with python2.6) but at least they do not actually get used in any of the generated scripts. So you get the zope.interface and zope.exceptions from the Zope2 tarball, which is good.

2. Keep allow-picked-versions = false. Add the two version pins. Now you know exactly which eggs you use without being surprised by sudden incompatible releases. But in the resulting scripts you are not using the zope.interface and zope.exceptions form the original Zope2 tarball. So subtle things might go wrong. BTW, it might be better to use some older eggs instead.

3. Avoid using any recipes that have dependencies on fake Zope 2 eggs.

Neither of the solutions is ideal I would say. With Plone 4 we use Zope 2.12 which is fully 'eggified', so no fake eggs are needed anymore, so this problem goes away then.

For Plone 3 (or earlier) I think we have to live with this problem and everyone will have to choose one of the partial solutions and be smart enough to be able to handle possible future problems.

Or does someone see a better, real solution?

Reactions are welcome as always via the contact form or via email or in this case via the Plone product-developers list.

Update: Note that the order in which the buildout parts get installed does not matter here. Buildout first gets all the recipes for all the parts, and if one of those recipes depends on zope.interface, it already goes wrong at that point. So that is even before the __init__ method of the recipes is called, let alone the install method.

Update 2: If I first explicitly do 'bin/buildout install zope2' and then 'bin/buildout' then it actually works. I thought I had tried that already. Still not ideal I would say, but it's a good extra option to have. A buildout run with the original buildout.cfg from above could then succeed in two steps:

$ rm -rf develop-eggs/ fake-eggs/ .installed.cfg parts/
$ bin/buildout
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface'.
Error: Picked: zope.interface = 3.6.1
$ bin/buildout install zope2
Installing zope2.
Creating fake eggs
$ bin/buildout
Updating zope2.
Updating fake eggs
Installing test.
Generated script '/Users/mauritsvanrees/tmp/pin/bin/test'.

Update 3: With these rather old version pins (from 2007) there are no dependencies on zope.interface and zope.exceptions anymore. It might be fine to use those, or you might miss some newer features and bug fixes:

zc.recipe.testrunner = 1.0.0
zope.testing = 3.5.1 

Thanks to Florian Schulze for some questions and remarks that got me thinking a bit further.

Packt Publishing has published a book by Juan Pablo Giménez and Marcos F. Romero: Plone 3.3 Products Development Cookbook. It is a very practical book. The authors do not present much theory, except when it is really needed to understand the recipes. With 70 recipes of about 5 pages each, there is not much room to go very deep into a subject. I think that does make for a book that I can point to in answer to questions on mailing lists: "Oh, just read that recipe in the cookbook on page 42."

Each recipe is structured like this, with some sections being optional:

Getting ready

some prerequisites for following the recipe.

How to do it...

actual steps you need to take, commands you need to perform in a terminal, lines you need to add in a file.

How it works...

explanations for what you just did and how Plone makes it work.

There's more...

this can point to more information, mostly online, or to alternative solutions.

See also:

this points to other recipes in the cookbook.

Sometimes the 'how to do it' steps make no sense without the hints in the 'how it works' section. For example, when 'how to do it' says this:

$ paster create -t plone3_buildout
$ cd pox

This is only going to work when you follow the instructions in 'how it works' where the authors tell you to enter 'pox' as the project name. In general, the distinction between these two sections does not always make sense to me. But I can imagine that the first time you follow a recipe you will want to read all sections, and the second time you only need the 'how to do it' section because you remember the other steps and explanations from the first time you did this; you may be happy then that this extra information is separated.

In the introduction, the authors say: "The book is for programmers who have some knowledge of Python, Plone, and Zope." Indeed without at least some prior experience you may get lost because you miss the bigger picture of the recipes. But the authors start by explaining how to install python and Plone on Linux and Windows, so they get you in a good starting position.

I would say the book is for beginning to intermediate Plone programmers. The only new information I saw for myself was about plone.app.content and dexterity. Still, it is good to have available when you think: "Today I want to bake a fresh portlet, let's get the cookbook." You will find good, solid information in this book.

As always, there is lots more info on the Packt website, like a table of contents and a sample chapter. You can get a very good idea of what the book is like and if this is something for you by reading the sample chapter about Creating a Custom Content Type with Paster.

Disclaimer: I got a review copy of this book for free. If you buy the cookbook via links on this page, I get a small fee.