Samenvatting

In dit rapport doe ik verslag van mijn afstudeeropdracht voor de studie Informatica aan de Hogeschool Rotterdam. Ik verbeter de eXtremeManagement tool: een project management tool gebaseerd op Extreme Programming principes en draaiend op het content management systeem Plone.

De Introductie (hoofdstuk 1) schetst de achtergrond van deze opdracht. Wat is Zope? Wat is Plone? Wat is eXtremeManagement? Hoe passen ze in elkaar?

Ik heb wat verbeteringen aangebracht in de gebruikersinterface (hoofdstuk 2). De meeste oorspronkelijke plannen werden niet uitgevoerd, om redenen uiteenlopend van simpel tijdgebrek tot nieuwe inzichten die het originele idee onderuit haalden. De focus van de opdracht verschoof daarmee naar het verbeteren van de kern, in plaats van het mooi maken van de buitenkant.

Plone 3.0 (hoofdstuk 3) vertelt het verhaal van het geschikt maken van eXtremeManagement voor de nieuwe versie 3.0 van Plone, die overigens nog de beta status heeft. Ik heb wat standaardaanpassingen doorgevoerd die alle Plone producten van derden moeten doen. Ik heb ook andere veranderingen gedaan die nodig bleken te zijn. Tenslotte heb ik een verbetering aan de kern van Plone bijgedragen om deze aanpassingen voor 3.0 simpeler te maken voor andere producten.

Met Zope 3 (hoofdstuk 4) komen we bij het hart van de zaak. Meer dan oorspronkelijk gedacht, moest de focus op dit deel gericht worden. Ik heb markeer- en functionele interfaces toegevoegd. Ik heb browser views gemaakt om een duidelijkere scheiding te brengen tussen data, presentatie en logica (Model-View-Controller). Ik heb annotaties ingebracht voor het bijhouden van inschattingen en gewerkte uren. Deze wijzigingen leveren samen een veel schonere versie van eXtremeManagement op dan er was bij de start van de afstudeeropdracht.

Ik trek de conclusie (hoofdstuk 5) dat eXtremeManagement schoon en toekomstklaar is en dat ik veel heb geleerd bij deze opdracht.

General actions

The general steps that every product needs while upgrading from Plone 2.5 to Plone 3.0 are described in the Upgrade guide on plone.org.

Some steps were already taken for Plone 2.5, like setting up the workflows with GenericSetup. Some steps are very simple. An example is that this:

get_transaction().commit(1)

needs to be replaced with:

transaction.commit(1)  

The other ones that needed checking in eXtremeManagement were:

• Searching users/groups via the Membership tool is deprecated; it turns out eXtremeManagement does not use any of the deprecated methods, so fifteen minutes of checking was all that was needed here.

• Portlets have a new infrastructure. Needed action: check that our classic portlets still work in Plone 3.0. They turn out to work fine. In the future we may want to investigate what the benefits of switching to the new portlet system are. A downside is that this would be incompatible with Plone 2.5, so it would require a separate branch.

• main_template.pt now uses Zope 3 viewlets. eXtremeManagement is not using an own version of that template, so no action was needed. In fact, we were not even using that template, which does not sound like the best idea. So I changed our templates to use the page layout as given by the main_template.

• The "Sharing" tab is now a global action. In previous Plone versions you had to explicitly add a Sharing action to your content type if you wanted to see the sharing tab when viewing that content in the Plone user interface. On the sharing tab you can share this content with others. Specifically in the case of eXtremeManagement: we have this tab on Projects, so we can give developers the Employee role and assign Tasks to them.

  On Plone 3.0 this action is now always available. This means that currently some of our content types actually list this tab twice, as they explicitly add it to themselves already. The problem now is that you cannot actually do this correctly on Plone 2.5 and 3.0 at the same time: either you have no sharing tab in 2.5 or you have two sharing tabs in 3.0. Since 3.0 is beta still, I opted to have it working correctly in Plone 2.5 only. Once Plone 3.0 is officially released, we can easily fix this on a branch. Or probably: at that point we can move 2.5 compatibility to its own branch and have trunk specifically for 3.0 support.

More fixes

There were some other steps that needed to be taken before eXtremeManagement would work on Plone 3.0. Some might need to be added to the general list and some may be specific for eXtremeManagement.

-  Finish
+  Finish

Introduction

Poi is an issue tracker for Plone. The code is in the collective.

The eXtremeManagement product from Zest Software and others depends on Poi. The biggest reason: Daniel Nouri has added a content type PoiTask to eXtremeManagement. You can add such a PoiTask to a Story and there easily link to one or more Poi issues that should be fixed. So I am interested in what happens with Poi and I occasionally add some code or run some tests on various Poi versions.

Poi has several branches. The ones that interest me here are 1.0 and plone3-support.

Poi has several dependencies: AddRemoveWidget, contentmigration, DataGridField and intelligenttext.

It can be annoying to have to hunt down where all dependencies can be downloaded and which version is needed. To make it easier on users, Poi has subversion bundles for this. The ones that interest me here are 10 and plone3. The bundles have in common that all dependencies are using trunk. So what are the differences? And with which Plone version can you use them?

Testing Poi

First of all: I improved the tests of Poi slightly, on all three branches that I investigate here. When you have an smtp server on your localhost, running the Poi tests would actually cause a few emails to be sent. They go to non-existing email addresses, so you (or some administrator) will get some mail delivery failures in your inbox, which is not nice. So in tests/ptc.py I replaced the default mail host with a simple mock mail host, which avoids this. That code was taken (and drastically simplified) from PasswordResetTool: thanks!

There is always one test that fails. This is a known issue: a deleted response causes stale SearchableText for an issue. If someone has an idea on how to fix this, that would be nice.

When below I say a branch or bundle is compatible with some Plone version then I mean that all tests except that one pass for that Plone version. The test combinations of Zope and Plone that I tried were:

• Plone 2.1.4 (tar ball) with Zope 2.8.9; for the tests I added PloneTestCase 0.8.2.
• Plone 2.5.3 (tar ball) with Zope 2.9.7
• Plone 3.0 (subversion trunk) with Zope 2.10.3

All were using python 2.4.

Note: I did not run any tests in combination with PloneSoftwareCenter.

instancemanager

I ran the tests with help from instancemanager. I use that tool a lot, so I am used to it, which helps. But I really like it for quickly running tests for (various versions of) a product on various versions of Plone. Here is my .instancemanager/poi.py file, ready for running the tests for the combination of Poi 1.0, Zope 2.9.7 and Plone 2.5.3. Commented out are the lines that are used for the other combinations:

python = 'python2.4'
#zope_version = '2.8.9'
zope_version = '2.9.7'
#zope_version = '2.10.3'

archive_sources = [
    #'PloneTestCase-0.8.2.tar.gz',
    ]
symlinkbundle_sources = [
    {'source': 'Poi10bundle',
     'url': 'http://svn.plone.org/svn/collective/Poi/bundles/10'},
    #{'source': 'Poitrunkbundle',
    # 'url': 'http://svn.plone.org/svn/collective/Poi/bundles/trunk'},
    #{'source': 'Poi30bundle',
    # 'url': 'http://svn.plone.org/svn/collective/Poi/bundles/plone3'},
    #{'url': 'http://svn.plone.org/svn/plone/bundles/3.0'},
    #{'url':'https://svn.plone.org/svn/plone/bundles/3.0-lib',
    # 'pylib': True},
    ]
archivebundle_sources = [
    #{'url': 'http://heanet.dl.sourceforge.net/sourceforge/plone/Plone-2.1.4.tar.gz'},
    {'url': 'http://plone.googlecode.com/files/Plone-2.5.3-final.tar.gz'},
    ]
main_products = ['Poi']

Then I just create an instance, fill the products directory and run the Poi tests with this single command:

$ instancemanager poi --create --products --test=MAIN

When the tests have run, I add and remove some comment signs to get the right versions and run that command again for the next combination.

Poi 10 bundle

• This has Poi branch 1.0. That branch was created by Daniel Nouri on 2007-05-17, before he added new functionality on trunk.
• Compatible with Plone 2.1 and 2.5

Poi trunk bundle

• This has Poi trunk, which has version number 1.1. Since putting 1.0 on a maintenance branch, trunk has seen the addition of link detection by Daniel (links are automatically created from #123 to issue 123 in this tracker; and r1234 points to revision/changelog 1234 if you specified a base url for that in your tracker). Plus some small changes by others that I did not investigate.
• Compatible with Plone 2.1 and 2.5.

Poi plone3 bundle

• This has Poi branch plone3-support. That branch was created by Alexander Limi on 2007-04-25. Goal is to add Plone 3 support.

• This bundle does not have intelligenttext. The reason is that intelligenttext is already available as a python module in Plone 3. It should be in your instance, in lib/python/plone/intelligenttext. The other versions of Poi throw this error, when they try to install intelligenttext:

  Products.CMFQuickInstallerTool.QuickInstallerTool.AlreadyInstalled: intelligenttext
  

  This python module is not exactly the same as the Product from the collective though. It misses some very recent additions there, like adding rel="nofollow" to links; this also means that the Poi tests had to be changed a bit. Or rather: I had to change the Poi tests on the other two branches to fit the intelligenttext changes.

  Update: Thomas Mueller (deepdiver) has updated plone.intelligenttext so it is in line with the collective product. He fixed the Poi tests accordingly. Thanks!

• Compatible with Plone 3.0 only.

Conclusions

• On Plone 2.1 or 2.5:
  • For a Poi version without sudden changes that would require a reinstall or a schema update: use branches/1.0 from bundles/10.
  • For recent improvements: use trunk from bundles/trunk.
• On Plone 3.0: use branches/plone3-support from bundles/plone3.

Introduction

I have a weblog. At the time of writing this is a hand-made Zope blog, consisting entirely of page templates and scripts, which sit in the Zope Database. See the status from early 2006 on the wayback machine. It works, it is fast, but it is kludgey. I have to do too many things before I can publish a new weblog entry. So there is room for improvement. Plus several other weblogs are available in Plone. Quills seems to be good, and I know a few persons who also use it, for instance my brother, so I choose to use that too.

Actually, I have two weblogs at my site. One is a general blog where I post personal things or entries relating to programming. The other is a completely Dutch blog for my church. Look at the old version. Actually it is more of a podcast. It contains small entries with a link to an external mp3 file containing a complete service of my church, and bigger entries with a link to a local mp3 file with just the sermon in case our pastor Erik was leading this service. So one service could actually have two entries. Again, it works, it is fast, but it is kludgey.

For this second blog, Quills seems like a reasonable choice as well. But to really make it useful, I would want two extra fields for the audio blog, that can contain links to those mp3 files. For the sermon it could be even nicer to be able to upload this file via Quills and have the link correctly made. But that may be added later. For now, I am happy with just two extra links in a Quills WeblogEntry.

Brave New Content Type

Alright, let's create a totally new content type, shall we? With ArchGenXML we can easily create a basic new content type, let's call it AudioEntry, that has the exact attributes that we need. Then again, other people have already thought a lot about what a weblog entry should contain and what it should do. We would introduce yet another weblog application, which Plone does not really need and that is currently only maintained by us. There are certainly occasions where this is a splendid option, but not here.

The olden days of Zope 2

Our second option is to extend Quills in the Zope 2 style: we create a class AudioEntry that inherits from the Quills WeblogEntry. If our class really offers something new, this is a fine choice. A good example is RichDocument. This extends the Schema of ATDocument to add a few attributes. The class inherits from ATDocument and so it uses the functions already defined for that basic ATDocument and adds a few functions that make sense for RichDocument. This is very well documented in Extending ATContentTypes by Martin Aspeli.

What would that mean for our use case of Extending Quills? AudioEntry would inherit from WeblogEntry and extend it with two attributes for the links. But it would not stop at that. For starters, a Quills Weblog can currently only contain Folders and WeblogEntries. Adding our AudioEntry there is not possible out of the box. Of course this is changeable, but then we might also have to override the Weblog class with our own AudioWeblog class. This can easily get out of hand and result in a cascade of overrides. Also, when someone else has a similar idea and introduces a VideoWeblogEntry, these two extension products would be incompatible.

There are certainly use cases like RichDocument where this Zope 2 way makes sense. But for our use case we simply want to add two links. Is there no easier and less intrusive way to do this?

Zope 3: A New Hope

When we extend a product in Zope 3 style, we adapt an existing content type without making a new content type. In our case this means that we only need to adapt WeblogEntry, without the need for changing the Weblog class. Also, we can arrange that our adaptations work for the VideoWeblogEntry class as well, as long as that class plays nice and implements the IWeblogEntry interface that Quills defines.

So already we can see that the Zope 3 way indeed brings new hope, in that our changes can be much less intrusive and much more compatible with other changes.

We will be using and introducing the following Zope 3 technologies:

• events
• utilities
• adapters
• annotations

General Strategy

We are going to make two products: one general product for extending an object when it fulfils some condition, and one specific product for extending a Quills WeblogEntry when it has audio content. The strategy for the general product is this:

• An event is handled by an event handler, which uses a...
• utility to decide if an...
• adapter is necessary, in which case...
• annotations are added to the original object.

So: when an event takes place, we ask a utility if this event warrants an adaptation of an object. The adaptation is implemented by adding annotations to that object.

This is implemented by the keywordannotator product. And this strategy probably has most people blinking their eyes and rereading that list. :) Don't panic. You probably do want to reread it, but if you do not fully understand it, you can just go to the next section, which will translate this general idea into a specific strategy for our use case. It should be much clearer then.

Quills Strategy

Do you remember the four Zope 3 technologies that we planned on using? Events, utilities, adapters and annotations. This is how our specific strategy uses those:

event

Someone adds a keyword audio to a WeblogEntry. When this happens, Zope fires a so-called IObjectModified event. We will register an event handler that springs into action when a WeblogEntry (or actually any object that claims to implement the IWeblogEntry interface) is modified or added.

utility

The code that handles this event, calls a utility. This utility looks at the WeblogEntry and decides that the addition of this keyword means that this entry now also implements the marker interface IMaudio that we will define. If an object implements or provides a 'normal' interface, then this object has all the functions and attributes that are defined by that interface. A marker interface has no functions or attributes. So it is basically just a label: the object is marked as being IMaudio.

We also make sure that we can later add annotations to this entry by letting it provide the standard IAttributeAnnotatable interface defined by Zope 3.

adapter

We can now actually adapt (extend) this object that implements IMaudio and add annotations to it. Adapting basically does the same that inheritance (the Zope 2 way) does, but with a different technique. You temporarily put a wrapper around an object, do something to it (in our case add annotations) and then remove the wrapper or just forget about it. Any other code (for example the code in Quills itself) does not know or care that the WeblogEntry object has been adapted or wrapped and now has something extra. To that code, the WeblogEntry is still a normal WeblogEntry, even though it now has annotations.

annotations

Annotations can be added in several ways, but most used is the method associated with the IAttributeAnnotatable interface that I mentioned above. In our case, two links are added in a new hidden attribute of that WeblogEntry.

This is implemented by the quadapter (Quills Adapter) product, which uses the keywordannotator product. In fact, this quadapter product is added in the examples directory of keywordannotator. By the way, both products have tests, so you can just run them and see for yourself that all this actually works.

This section is rather important. If you understand what our goal and our strategy is, you stand a good chance of understanding the next sections which present the actual code. On the other hand, if you prefer a bottom up approach, then reading the next sections may help you in understanding this strategy.

Each of the next few sections first present the code from the general keywordannotator product and then the code from the specific quadapter product. Sometimes I find it hard myself to follow what keywordannotator is actually doing, but reading the corresponding quadapter code usually helps. :)

Event handler

def annotationEventHandler(ob, event):
    from zope.component import getUtility
    decider = getUtility(IAnnotationDecider, context=ob)
    if decider.matchesKeywords(ob):
        decider.provideInterfaces(ob)

Utility

class DefaultAnnotationDecider(object):
    implements(IAnnotationDecider)
    keywords = KEYWORDS
    ifaces = (IKeywordMatch,)
    def matchesKeywords(self, object):
        ...
    def provideInterfaces(self, object):
        for iface in self.ifaces:
            if not iface.providedBy(object):
                alsoProvides(object, iface)

class AudioDecider(DefaultAnnotationDecider):
    implements(IAnnotationDecider)
    keywords = KEYWORDS
    ifaces = PROVIDE_INTERFACES

KEYWORDS = ['audio', 'preken']
PROVIDE_INTERFACES = (IMaudio, IAttributeAnnotatable,)

Current Status

We have covered a lot of ground already. So before we continue with adapters and annotations, let's take a break and see what we have accomplished so far.

If you are only using the keywordannotator product, the situation is this:

• An object implementing the IAttributeAnnotatable interface,
• given the special keyword ('special'),
• now also implements the IKeywordMatch interface.

If next to keywordannotator you are also using Quills and the quadapter product, then your situation is this:

• An object implementing the IWeblogEntry interface,
• given one of the special keywords ('audio' or 'preken'),
• now also implements the IMaudio interface
• and the IAttributeAnnotatable interface.

If this is not clear yet, then it is better to reread some of the previous sections. If you do understand the current status, then it is time for adapters and annotations.

Adapter

class IAudioAnnotations(Interface):
    """Provide access to the audio annotations of an IMaudio object.
    """

    completeURL = schema.URI(title=u'URL to complete audio content')
    partURL = schema.URI(title=u'URL to a part of the audio content')

class KeywordBasedAnnotations(object):
    ...
    def __init__(self, context):
        self.context = context
        annotations = IAnnotations(self.context)
        self._metadata = annotations.get(self._anno_key, None)
        if self._metadata is None:
            self._metadata = PersistentDict()
            annotations[self._anno_key] = self._metadata

annotations = IAnnotations(self.context)

class AudioAnnotations(KeywordBasedAnnotations):
    ...
    def __get_completeURL(self):
        return self._metadata.get(COMP_ANNO)
    def __set_completeURL(self, url):
        self._metadata[COMP_ANNO] = url
    completeURL = property(__get_completeURL, __set_completeURL)

Annotations

In the previous section we have actually already seen how annotations are added to an object and that they are basically just a python dictionary. But before this is possible, we need to add one line to the configure.zcml of keywordannotator:

This directive loads another zcml directive which is in the zope software directory, in the file lib/python/zope/app/annotation/configure.zcml:

This actually concludes our strategy. All the code and configuration is now in place to add those annotations to a WeblogEntry by adapting it if it implements or provides the IMaudio interface, which is added to it by a utility if if decides that the condition is met after an event takes places that is handled by an event handler. If you understood what I just wrote, then you are very smart. :)

Adding Annotations

At this point, if you look at the state of things on the Plone level, we can add a normal WeblogEntry and give it a keyword 'audio'. Then our code makes sure that it provides the IMaudio interface. And then it stops! There is not yet a way to actually put something in the annotations. But we can arrange that. First we install a new product: CMFonFive.

That product may get assimilated into the core of CMF itself some day (Plone is based on CMF, the Content Management Framework). But for now this extra product is needed for the following lines that we add in the configure.zcml of quadapter:

This adds a menu item in the Plone site to the object tabs of any object that implements the IMaudio interface. When you add this code, restart your Zope instance, and look at a WeblogEntry that has one of the special words and thus provides the IMaudio interface, you will see a tab that links to the action maudio_edit.

At that point we are almost done. We now need to make an edit form for that tab, that calls the code that sets the Annotations for this object. If you know how to make an edit form in html, then you should be able to make this yourself, possibly with the use of the Archetypes product.

Viewing Annotations

When you have added those links in the annotation of a WeblogEntry, you also want to be able to see them when you actually look at a WeblogEntry in your browser. We can use one more Zope 3 technique for this: the BrowserView. Essentially this also is an adapter. We need to define a BrowserView in quadapter:

This basically means that for an object providing the IMaudio interface we have a python class that gives us some functions to call in an html page template. In fact, we can now copy the file entry_macros.pt from Quills and add these lines at the right spot:

This uses a maudio_macros.pt file that gets the links from the object. Now when you view a normal WeblogEntry you just see the normal page that you would otherwise see. But when you view an entry with the IMaudio interface, for which you have added links with the edit form in the previous section, you will now see some extra text containing those links. The specifics are left as an exercise to the reader as they are just standard page template techniques, which should be familiar and which are not too interesting for this tutorial. If you do not get it working, contact me. Well, okay, I have just added them in the skins folder of quadapter.

Conclusions

Using Zope 3 like this is:

• clean:
  • Only one page template from Quills is changed and no code is overwritten.
  • If we added a metal:macro with a fill-slot to Quills, even that single overwrite would not be necessary.
• compatible:
  • We only add features, we do not change existing behaviour of WeblogEntries.
  • Our audio enhanced WeblogEntries are still first class WeblogEntries. Existing code that expects a normal WeblogEntry can handle our audio entries just fine.
  • Our changes work on any content type that implements the IWeblogEntry interface. So if someone really does create a VideoWeblogEntry implementing IWeblogEntry, our code would work for such an object as well.

In other words: Zope 3 is the place to be!