Introduction

Poi is an issue tracker for Plone. The code is in the collective.

The eXtremeManagement product from Zest Software and others depends on Poi. The biggest reason: Daniel Nouri has added a content type PoiTask to eXtremeManagement. You can add such a PoiTask to a Story and there easily link to one or more Poi issues that should be fixed. So I am interested in what happens with Poi and I occasionally add some code or run some tests on various Poi versions.

Poi has several branches. The ones that interest me here are 1.0 and plone3-support.

Poi has several dependencies: AddRemoveWidget, contentmigration, DataGridField and intelligenttext.

It can be annoying to have to hunt down where all dependencies can be downloaded and which version is needed. To make it easier on users, Poi has subversion bundles for this. The ones that interest me here are 10 and plone3. The bundles have in common that all dependencies are using trunk. So what are the differences? And with which Plone version can you use them?

Testing Poi

First of all: I improved the tests of Poi slightly, on all three branches that I investigate here. When you have an smtp server on your localhost, running the Poi tests would actually cause a few emails to be sent. They go to non-existing email addresses, so you (or some administrator) will get some mail delivery failures in your inbox, which is not nice. So in tests/ptc.py I replaced the default mail host with a simple mock mail host, which avoids this. That code was taken (and drastically simplified) from PasswordResetTool: thanks!

There is always one test that fails. This is a known issue: a deleted response causes stale SearchableText for an issue. If someone has an idea on how to fix this, that would be nice.

When below I say a branch or bundle is compatible with some Plone version then I mean that all tests except that one pass for that Plone version. The test combinations of Zope and Plone that I tried were:

• Plone 2.1.4 (tar ball) with Zope 2.8.9; for the tests I added PloneTestCase 0.8.2.
• Plone 2.5.3 (tar ball) with Zope 2.9.7
• Plone 3.0 (subversion trunk) with Zope 2.10.3

All were using python 2.4.

Note: I did not run any tests in combination with PloneSoftwareCenter.

instancemanager

I ran the tests with help from instancemanager. I use that tool a lot, so I am used to it, which helps. But I really like it for quickly running tests for (various versions of) a product on various versions of Plone. Here is my .instancemanager/poi.py file, ready for running the tests for the combination of Poi 1.0, Zope 2.9.7 and Plone 2.5.3. Commented out are the lines that are used for the other combinations:

python = 'python2.4'
#zope_version = '2.8.9'
zope_version = '2.9.7'
#zope_version = '2.10.3'

archive_sources = [
    #'PloneTestCase-0.8.2.tar.gz',
    ]
symlinkbundle_sources = [
    {'source': 'Poi10bundle',
     'url': 'http://svn.plone.org/svn/collective/Poi/bundles/10'},
    #{'source': 'Poitrunkbundle',
    # 'url': 'http://svn.plone.org/svn/collective/Poi/bundles/trunk'},
    #{'source': 'Poi30bundle',
    # 'url': 'http://svn.plone.org/svn/collective/Poi/bundles/plone3'},
    #{'url': 'http://svn.plone.org/svn/plone/bundles/3.0'},
    #{'url':'https://svn.plone.org/svn/plone/bundles/3.0-lib',
    # 'pylib': True},
    ]
archivebundle_sources = [
    #{'url': 'http://heanet.dl.sourceforge.net/sourceforge/plone/Plone-2.1.4.tar.gz'},
    {'url': 'http://plone.googlecode.com/files/Plone-2.5.3-final.tar.gz'},
    ]
main_products = ['Poi']

Then I just create an instance, fill the products directory and run the Poi tests with this single command:

$ instancemanager poi --create --products --test=MAIN

When the tests have run, I add and remove some comment signs to get the right versions and run that command again for the next combination.

Poi 10 bundle

• This has Poi branch 1.0. That branch was created by Daniel Nouri on 2007-05-17, before he added new functionality on trunk.
• Compatible with Plone 2.1 and 2.5

Poi trunk bundle

• This has Poi trunk, which has version number 1.1. Since putting 1.0 on a maintenance branch, trunk has seen the addition of link detection by Daniel (links are automatically created from #123 to issue 123 in this tracker; and r1234 points to revision/changelog 1234 if you specified a base url for that in your tracker). Plus some small changes by others that I did not investigate.
• Compatible with Plone 2.1 and 2.5.

Poi plone3 bundle

• This has Poi branch plone3-support. That branch was created by Alexander Limi on 2007-04-25. Goal is to add Plone 3 support.

• This bundle does not have intelligenttext. The reason is that intelligenttext is already available as a python module in Plone 3. It should be in your instance, in lib/python/plone/intelligenttext. The other versions of Poi throw this error, when they try to install intelligenttext:

  Products.CMFQuickInstallerTool.QuickInstallerTool.AlreadyInstalled: intelligenttext
  

  This python module is not exactly the same as the Product from the collective though. It misses some very recent additions there, like adding rel="nofollow" to links; this also means that the Poi tests had to be changed a bit. Or rather: I had to change the Poi tests on the other two branches to fit the intelligenttext changes.

  Update: Thomas Mueller (deepdiver) has updated plone.intelligenttext so it is in line with the collective product. He fixed the Poi tests accordingly. Thanks!

• Compatible with Plone 3.0 only.

Conclusions

• On Plone 2.1 or 2.5:
  • For a Poi version without sudden changes that would require a reinstall or a schema update: use branches/1.0 from bundles/10.
  • For recent improvements: use trunk from bundles/trunk.
• On Plone 3.0: use branches/plone3-support from bundles/plone3.

Introduction

I have a weblog. At the time of writing this is a hand-made Zope blog, consisting entirely of page templates and scripts, which sit in the Zope Database. See the status from early 2006 on the wayback machine. It works, it is fast, but it is kludgey. I have to do too many things before I can publish a new weblog entry. So there is room for improvement. Plus several other weblogs are available in Plone. Quills seems to be good, and I know a few persons who also use it, for instance my brother, so I choose to use that too.

Actually, I have two weblogs at my site. One is a general blog where I post personal things or entries relating to programming. The other is a completely Dutch blog for my church. Look at the old version. Actually it is more of a podcast. It contains small entries with a link to an external mp3 file containing a complete service of my church, and bigger entries with a link to a local mp3 file with just the sermon in case our pastor Erik was leading this service. So one service could actually have two entries. Again, it works, it is fast, but it is kludgey.

For this second blog, Quills seems like a reasonable choice as well. But to really make it useful, I would want two extra fields for the audio blog, that can contain links to those mp3 files. For the sermon it could be even nicer to be able to upload this file via Quills and have the link correctly made. But that may be added later. For now, I am happy with just two extra links in a Quills WeblogEntry.

Brave New Content Type

Alright, let's create a totally new content type, shall we? With ArchGenXML we can easily create a basic new content type, let's call it AudioEntry, that has the exact attributes that we need. Then again, other people have already thought a lot about what a weblog entry should contain and what it should do. We would introduce yet another weblog application, which Plone does not really need and that is currently only maintained by us. There are certainly occasions where this is a splendid option, but not here.

The olden days of Zope 2

Our second option is to extend Quills in the Zope 2 style: we create a class AudioEntry that inherits from the Quills WeblogEntry. If our class really offers something new, this is a fine choice. A good example is RichDocument. This extends the Schema of ATDocument to add a few attributes. The class inherits from ATDocument and so it uses the functions already defined for that basic ATDocument and adds a few functions that make sense for RichDocument. This is very well documented in Extending ATContentTypes by Martin Aspeli.

What would that mean for our use case of Extending Quills? AudioEntry would inherit from WeblogEntry and extend it with two attributes for the links. But it would not stop at that. For starters, a Quills Weblog can currently only contain Folders and WeblogEntries. Adding our AudioEntry there is not possible out of the box. Of course this is changeable, but then we might also have to override the Weblog class with our own AudioWeblog class. This can easily get out of hand and result in a cascade of overrides. Also, when someone else has a similar idea and introduces a VideoWeblogEntry, these two extension products would be incompatible.

There are certainly use cases like RichDocument where this Zope 2 way makes sense. But for our use case we simply want to add two links. Is there no easier and less intrusive way to do this?

Zope 3: A New Hope

When we extend a product in Zope 3 style, we adapt an existing content type without making a new content type. In our case this means that we only need to adapt WeblogEntry, without the need for changing the Weblog class. Also, we can arrange that our adaptations work for the VideoWeblogEntry class as well, as long as that class plays nice and implements the IWeblogEntry interface that Quills defines.

So already we can see that the Zope 3 way indeed brings new hope, in that our changes can be much less intrusive and much more compatible with other changes.

We will be using and introducing the following Zope 3 technologies:

• events
• utilities
• adapters
• annotations

General Strategy

We are going to make two products: one general product for extending an object when it fulfils some condition, and one specific product for extending a Quills WeblogEntry when it has audio content. The strategy for the general product is this:

• An event is handled by an event handler, which uses a...
• utility to decide if an...
• adapter is necessary, in which case...
• annotations are added to the original object.

So: when an event takes place, we ask a utility if this event warrants an adaptation of an object. The adaptation is implemented by adding annotations to that object.

This is implemented by the keywordannotator product. And this strategy probably has most people blinking their eyes and rereading that list. :) Don't panic. You probably do want to reread it, but if you do not fully understand it, you can just go to the next section, which will translate this general idea into a specific strategy for our use case. It should be much clearer then.

Quills Strategy

Do you remember the four Zope 3 technologies that we planned on using? Events, utilities, adapters and annotations. This is how our specific strategy uses those:

event

Someone adds a keyword audio to a WeblogEntry. When this happens, Zope fires a so-called IObjectModified event. We will register an event handler that springs into action when a WeblogEntry (or actually any object that claims to implement the IWeblogEntry interface) is modified or added.

utility

The code that handles this event, calls a utility. This utility looks at the WeblogEntry and decides that the addition of this keyword means that this entry now also implements the marker interface IMaudio that we will define. If an object implements or provides a 'normal' interface, then this object has all the functions and attributes that are defined by that interface. A marker interface has no functions or attributes. So it is basically just a label: the object is marked as being IMaudio.

We also make sure that we can later add annotations to this entry by letting it provide the standard IAttributeAnnotatable interface defined by Zope 3.

adapter

We can now actually adapt (extend) this object that implements IMaudio and add annotations to it. Adapting basically does the same that inheritance (the Zope 2 way) does, but with a different technique. You temporarily put a wrapper around an object, do something to it (in our case add annotations) and then remove the wrapper or just forget about it. Any other code (for example the code in Quills itself) does not know or care that the WeblogEntry object has been adapted or wrapped and now has something extra. To that code, the WeblogEntry is still a normal WeblogEntry, even though it now has annotations.

annotations

Annotations can be added in several ways, but most used is the method associated with the IAttributeAnnotatable interface that I mentioned above. In our case, two links are added in a new hidden attribute of that WeblogEntry.

This is implemented by the quadapter (Quills Adapter) product, which uses the keywordannotator product. In fact, this quadapter product is added in the examples directory of keywordannotator. By the way, both products have tests, so you can just run them and see for yourself that all this actually works.

This section is rather important. If you understand what our goal and our strategy is, you stand a good chance of understanding the next sections which present the actual code. On the other hand, if you prefer a bottom up approach, then reading the next sections may help you in understanding this strategy.

Each of the next few sections first present the code from the general keywordannotator product and then the code from the specific quadapter product. Sometimes I find it hard myself to follow what keywordannotator is actually doing, but reading the corresponding quadapter code usually helps. :)

Event handler

def annotationEventHandler(ob, event):
    from zope.component import getUtility
    decider = getUtility(IAnnotationDecider, context=ob)
    if decider.matchesKeywords(ob):
        decider.provideInterfaces(ob)

Utility

class DefaultAnnotationDecider(object):
    implements(IAnnotationDecider)
    keywords = KEYWORDS
    ifaces = (IKeywordMatch,)
    def matchesKeywords(self, object):
        ...
    def provideInterfaces(self, object):
        for iface in self.ifaces:
            if not iface.providedBy(object):
                alsoProvides(object, iface)

class AudioDecider(DefaultAnnotationDecider):
    implements(IAnnotationDecider)
    keywords = KEYWORDS
    ifaces = PROVIDE_INTERFACES

KEYWORDS = ['audio', 'preken']
PROVIDE_INTERFACES = (IMaudio, IAttributeAnnotatable,)

Current Status

We have covered a lot of ground already. So before we continue with adapters and annotations, let's take a break and see what we have accomplished so far.

If you are only using the keywordannotator product, the situation is this:

• An object implementing the IAttributeAnnotatable interface,
• given the special keyword ('special'),
• now also implements the IKeywordMatch interface.

If next to keywordannotator you are also using Quills and the quadapter product, then your situation is this:

• An object implementing the IWeblogEntry interface,
• given one of the special keywords ('audio' or 'preken'),
• now also implements the IMaudio interface
• and the IAttributeAnnotatable interface.

If this is not clear yet, then it is better to reread some of the previous sections. If you do understand the current status, then it is time for adapters and annotations.

Adapter

class IAudioAnnotations(Interface):
    """Provide access to the audio annotations of an IMaudio object.
    """

    completeURL = schema.URI(title=u'URL to complete audio content')
    partURL = schema.URI(title=u'URL to a part of the audio content')

class KeywordBasedAnnotations(object):
    ...
    def __init__(self, context):
        self.context = context
        annotations = IAnnotations(self.context)
        self._metadata = annotations.get(self._anno_key, None)
        if self._metadata is None:
            self._metadata = PersistentDict()
            annotations[self._anno_key] = self._metadata

annotations = IAnnotations(self.context)

class AudioAnnotations(KeywordBasedAnnotations):
    ...
    def __get_completeURL(self):
        return self._metadata.get(COMP_ANNO)
    def __set_completeURL(self, url):
        self._metadata[COMP_ANNO] = url
    completeURL = property(__get_completeURL, __set_completeURL)

Annotations

In the previous section we have actually already seen how annotations are added to an object and that they are basically just a python dictionary. But before this is possible, we need to add one line to the configure.zcml of keywordannotator:

This directive loads another zcml directive which is in the zope software directory, in the file lib/python/zope/app/annotation/configure.zcml:

This actually concludes our strategy. All the code and configuration is now in place to add those annotations to a WeblogEntry by adapting it if it implements or provides the IMaudio interface, which is added to it by a utility if if decides that the condition is met after an event takes places that is handled by an event handler. If you understood what I just wrote, then you are very smart. :)

Adding Annotations

At this point, if you look at the state of things on the Plone level, we can add a normal WeblogEntry and give it a keyword 'audio'. Then our code makes sure that it provides the IMaudio interface. And then it stops! There is not yet a way to actually put something in the annotations. But we can arrange that. First we install a new product: CMFonFive.

That product may get assimilated into the core of CMF itself some day (Plone is based on CMF, the Content Management Framework). But for now this extra product is needed for the following lines that we add in the configure.zcml of quadapter:

This adds a menu item in the Plone site to the object tabs of any object that implements the IMaudio interface. When you add this code, restart your Zope instance, and look at a WeblogEntry that has one of the special words and thus provides the IMaudio interface, you will see a tab that links to the action maudio_edit.

At that point we are almost done. We now need to make an edit form for that tab, that calls the code that sets the Annotations for this object. If you know how to make an edit form in html, then you should be able to make this yourself, possibly with the use of the Archetypes product.

Viewing Annotations

When you have added those links in the annotation of a WeblogEntry, you also want to be able to see them when you actually look at a WeblogEntry in your browser. We can use one more Zope 3 technique for this: the BrowserView. Essentially this also is an adapter. We need to define a BrowserView in quadapter:

This basically means that for an object providing the IMaudio interface we have a python class that gives us some functions to call in an html page template. In fact, we can now copy the file entry_macros.pt from Quills and add these lines at the right spot:

This uses a maudio_macros.pt file that gets the links from the object. Now when you view a normal WeblogEntry you just see the normal page that you would otherwise see. But when you view an entry with the IMaudio interface, for which you have added links with the edit form in the previous section, you will now see some extra text containing those links. The specifics are left as an exercise to the reader as they are just standard page template techniques, which should be familiar and which are not too interesting for this tutorial. If you do not get it working, contact me. Well, okay, I have just added them in the skins folder of quadapter.

Conclusions

Using Zope 3 like this is:

• clean:
  • Only one page template from Quills is changed and no code is overwritten.
  • If we added a metal:macro with a fill-slot to Quills, even that single overwrite would not be necessary.
• compatible:
  • We only add features, we do not change existing behaviour of WeblogEntries.
  • Our audio enhanced WeblogEntries are still first class WeblogEntries. Existing code that expects a normal WeblogEntry can handle our audio entries just fine.
  • Our changes work on any content type that implements the IWeblogEntry interface. So if someone really does create a VideoWeblogEntry implementing IWeblogEntry, our code would work for such an object as well.

In other words: Zope 3 is the place to be!

Adding content in a different position

This product uses a roulation policy when adding content to a folder. You can let content types implement the IRoulationContent interface. If you do not understand what I just said, ask a programmer. ;-) When such a content object is added to a folder, the item now gets added right below the top. This means, not at position zero (which is the top) but position one.

Roulating content to the bottom

A container (folder) can implement the IRoulationContainer interface. This means that every day the top most content object in that container is moved to the bottom.

Dependencies

• Zope 2.9 (or higher). The product uses zope 3 events, which means that it likely will not work with Zope 2.8.
• Archetypes (for the moveObjects* functions). I use the Plone 2.5 bundle containing that.
• For testing:
  • Five: at least version 1.4 as that contains testbrowser.
  • ATConttentTypes as we will use ATDocument and ATFolder to demonstrate our product. I use the Plone 2.5 bundle containing that.

Developers

See roulation.txt as a doctest on how to use this product. But basically it is this.

Choose content types that will implement our interfaces. For example, ATDocument can be roulation content.

>>> from zope.interface import classImplements
>>> from Products.roulationpolicy.interfaces import IRoulationContent
>>> from Products.ATContentTypes.content.document import ATDocument
>>> classImplements(ATDocument, IRoulationContent)

This will make sure that new ATDocuments will be added right below the top of a folder.

Now you can let a folderish content type be a roulation container, for example ATFolder.

>>> from Products.roulationpolicy.interfaces import IRoulationContainer
>>> from Products.ATContentTypes.content.folder import ATFolder
>>> classImplements(ATFolder, IRoulationContainer)

Now add some code in a place that is locical to you that fires an event when the items in the folderish type need to be roulated:

>>> from zope.event import notify
>>> from Products.roulationpolicy.events import ContainerRoulationEvent
>>> notify(ContainerRoulationEvent(self.folder))

This will make sure that the top most item in self.folder gets moved to the bottom, making all the other items shift to the top.

Instead, you can also call a browser view that does this for you

>>> from Products.Five.testbrowser import Browser
>>> browser = Browser()
>>> browser.open(self.folder.absolute_url() + '/@@roulate_folder')

Reasons

• This way we can get rid of several kludgey methods on our content types, like manage_afterAdd and reindexObject. Using events, like already started on trunk, also helps here.

• We get rid of ComputedFields like getRawActualHours. On a Booking this was okay: it just changed 1 hours plus 30 minutes into 1.5 hours; this is still done actually, but with a small twist. But on a Task this would calculate the total of getRawActualHours of all its children, either by waking up these objects or by querying the catalog. Instead, with annotations and events we now make sure that this total is only calculated when a Booking has been added or changed or moved.

• During a reinstall GenericSetup sees that we have some indexes in our catalog.xml so it unhelpfully empties those indexes. So we need to reindex them during our reinstall. This always seemed to have a negative effect on the consistency of booked hours and estimates in the catalog. At least it seemed to help to manually recatalog first the Bookings, then the Tasks and then the Stories and possibly the Iterations after a reinstall.

  GenericSetup only empties the indexes though and not the metadata columns, so this should not have been necessary. Maybe I was misled through coincidences into believing that this helped. Anyway, now everything is stored on the object itself or in annotations, so the order in which you recatalog the content types really does not matter anymore. :)

• Our content types are simpler now.

  • A Task does not need to be aware of any changes to its Bookings and it does not need to inform its parent Story that its estimate has changed. Keeping everything updated is now the responsibility of the zope 3 events.
  • Information about children (total hours booked or estimated) is now stored separately in annotations.

In general, this approach makes me feel sane again. Plus it allows me to play with annotations and they are cool. ;-)

Implementation

• The annotations are taken care of in the timing/ directory.

• A few interfaces have been introduced there:

  • IActualHours: this is for content types that wish to store actual hours. Bookings actually implement this functionality already, so when you adapt a Booking to the IActualHours interface the Booking itself is returned.
  • IActualHoursContainer: This is a marker interface that says: this object has children with actual hours. Iterations, Stories and (Poi)Tasks are marked with this. Actually, marker interface may be a bad choice of words as it does have one method that needs to be provided: contentValues. Since the types mentioned are folderish types they provide this method already. Objects providing this interface should also be adaptable to IActualHours.
  • IEstimate: this is for content types that wish to store estimates. (Poi)Tasks actually implement this functionality already, so when you adapt a Task to the IEstimate interface the Task itself is returned.
  • IEstimateContainer: This is a marker interface that says: this object has children with estimates. Iterations and Stories are marked with this. The same remarks as with IActualHoursContainer can be made. Objects providing this interface should also be adaptable to IEstimate.

• By adapting an object to IActualHours or IEstimate you get an object that has a recalc method that recalculates its actual hours or the estimate and stores that in an attribute. For the Containers this is stored in annotations.

• For Booking this is stored in a property that functions as a computed field. The recalc method then only needs to reindex the object:

  @property
  def actual_time(self):
      return self.getHours() + (self.getMinutes() / 60.0)
  
  security.declarePublic('recalc')
  def recalc(self):
      self.reindexObject(idxs=['actual_time'])
  

• The same is done for the estimate of Tasks.

• For the various Containers the recalc method fetches its children. It does not matter if they are Bookings, Tasks or Stories. It then adapts them to the IActualHours or IEstimate interface and adds the value of the stored property to its own total. For IActualHoursContainers it looks like this:

  def recalc(self):
      """Recalculate the total booked hours for this container.
      """
      context = self.context
      total = 0.0
      for obj in context.contentValues():
          actual = IActualHours(obj, None)
          if actual is not None:
              total += actual.actual_time
      self.actual_time = total
      context.reindexObject(idxs=['actual_time'])
  

• Zope 3 events are used to arrange that everything is updated at the right time, namely when a relevant object is added, moved, removed or changed. Event handlers make sure that the value for the object itself is recalculated and that its parents, grandparent, etcetera are also recalculated. This uses the following code, where adapter is either IActualHours or IEstimate:

  def recursivelyRecalc(object, adapter):
      current = aq_inner(object)
      anno = adapter(current, None)
      while anno is not None:
          anno.recalc()
          current = aq_parent(current)
          anno = adapter(current, None)
  

• This value is also put in the catalog:

  from Products.CMFPlone import CatalogTool as catalogtool
  
  def actual(object, portal, **kw):
      anno = IActualHours(object, None)
      if anno is not None:
          return anno.actual_time
      return None
  
  catalogtool.registerIndexableAttribute('actual_time', actual)
  

Extra cleanup

While I was busy, I did some related cleanup:

• Indexes and metadata for the catalog are no longer registered in the code for the content types themselves (so the content/ directory). This is now in the GenericSetup profile in catalog.xml. The former code might not even work anymore in newer versions of Plone or Archetypes, if I am informed correctly.
• In the course of time we registered far too many indexes and metadata for our content types. Most indexes were unneeded and have been removed, except getAssignees and getBookingDate. By reinstalling or applying the GenericSetup profile the others will be removed if you use Plone 2.5.3 (GenericSetup 1.2). Removing metadata columns from the catalog is not automatically possibly currently with GenericSetup. I submitted a patch [1] for that. But the following are unused and you can manually remove them: getRawActualHours, getRawDifference, getRawEstimate. Two of those have simply gotten a different name though: actual_time and estimate. The new names feel more pythonic. :) It seemed logical to change the names while I was at it.
• When you migrate the schema of Tasks then under the hood all your Tasks are created fresh. This means that lots of emails will be sent out to inform people that some tasks have been assigned to them. This is bad and I apologize for the times in the past when this went wrong while I was testing and I ended up spamming my colleagues... :) Our reinstall actually knows this and works around it by setting a boolean value that tells Tasks to not send those emails. This functionality has been there for a while, but you can now also flip this switch manually: in the portal_properties tool there is a property sheet xm_properties with a setting send_task_mails that controls this.