Basic use of catalog.xml

Using GenericSetup you can add indexes and metadata columns to the portal_catalog with a catalog.xml file like this:

<!--?xml version="1.0"?--> <object name="portal_catalog" width="300" height="150">
<index name="getSomething" meta_type="KeywordIndex">
<indexed_attr value="getSomething"></indexed_attr>
</index>
<column value="getSomething"></column></object>

Specifying an index will add an index for getSomething in the portal_catalog so you can search on it. Specifying a column will add getSomething to the metadata of the catalog brains, so you can ask a brain what his value is for getSomething. These are very different use cases, so before you add both an index and a column you may want to think if you really need them both or if one of them is enough.

Anyway, specifying a column here is fine. Nothing wrong with it. Do note that when you add a column here this does not make getSomething available in the current brains in the catalog. You will need to do a reindex; a clear and rebuild of the catalog would do it, but it may be enough to find and reindex items of one specific content type that has this field. Depending on your specific situation this may or may not be an issue.

What happens with indexes?

What is almost never a good idea however, is specifying an index here. What this does is it creates the index in the portal_catalog. The index is not filled automatically, so you will have to reindex it manually (or write some code for that). But what happens the next time you reinstall your product or reapply your profile? The index gets removed and recreated. So the index is empty and you will need to reindex it manually again! That is not very handy.

This might be fixable in the GenericSetup import handler for catalog.xml. But this is hard to do as it is currently not possible to verify without a doubt that the index that is currently in the portal_catalog has the same configuration as specified in the catalog.xml. For example, the id might be the same but the existing index might be a FieldIndex and catalog.xml might specify a KeywordIndex. This specific check might be doable, but there are other indexes for which this is not so simple.

Import handler

So, what do you do instead? You add an import handler. I have done that in several products, so instead of copy-pasting code from one of those products I might as well copy-paste it from my weblog. :-)

Write an import step in setuphandlers.py:

import logging
from Products.CMFCore.utils import getToolByName
# The profile id of your package:
PROFILE_ID = 'profile-your.product:default'


def add_catalog_indexes(context, logger=None):
    """Method to add our wanted indexes to the portal_catalog.

    @parameters:

    When called from the import_various method below, 'context' is
    the plone site and 'logger' is the portal_setup logger.  But
    this method can also be used as upgrade step, in which case
    'context' will be portal_setup and 'logger' will be None.
    """
    if logger is None:
        # Called as upgrade step: define our own logger.
        logger = logging.getLogger('your.package')

    # Run the catalog.xml step as that may have defined new metadata
    # columns.  We could instead add  to
    # the registration of our import step in zcml, but doing it in
    # code makes this method usable as upgrade step as well.  Note that
    # this silently does nothing when there is no catalog.xml, so it                                                                                  
    # is quite safe.
    setup = getToolByName(context, 'portal_setup')
    setup.runImportStepFromProfile(PROFILE_ID, 'catalog')

    catalog = getToolByName(context, 'portal_catalog')
    indexes = catalog.indexes()
    # Specify the indexes you want, with ('index_name', 'index_type')
    wanted = (('getSomething', 'FieldIndex'),
              ('getAnother', 'KeywordIndex'),
              )
    indexables = []
    for name, meta_type in wanted:
        if name not in indexes:
            catalog.addIndex(name, meta_type)
            indexables.append(name)
            logger.info("Added %s for field %s.", meta_type, name)
    if len(indexables) > 0:
        logger.info("Indexing new indexes %s.", ', '.join(indexables))
        catalog.manage_reindexIndex(ids=indexables)


def import_various(context):
    """Import step for configuration that is not handled in xml files.
    """
    # Only run step if a flag file is present
    if context.readDataFile('your_package-default.txt') is None:
        return
    logger = context.getLogger('your.package')
    site = context.getSite()
    add_catalog_indexes(site, logger)

If you need to replace an existing FieldIndex with a KeywordIndex this code is not enough, but we ignore that possibility here.

The rest should be nothing new, but let's make it clear and explicit by showing everything here.

Register your GenericSetup code

I usually end up moving the registration of GenericSetup profiles, import and upgrade steps in a separate zcml file called profiles.zcml. We need to include that in our configure.zcml:

We register our profile and our steps in profiles.zcml:

metadata.xml

Create a profiles/default directory if this does not exist yet. This must have a metadata.xml file like this:

 

  1001

The version number should be an integer. This profile version has nothing at all to do with the number in our version.txt or setup.py, but that is a different discussion. The destination number in our last upgrade step registration must match this metadata version.

Flag file

When you apply a GenericSetup profile or (re)install a product, every import step defined by any package is called. One step looks for a catalog.xml file within the profile directory of the profile that is being applied and exits if it is not there; another looks for a skins.xml and exits if it is not there. Our own import step must do the same, otherwise our code is executed far too often, even when our product is not installed.

As seen above, our import_various import handler starts with this check:

if context.readDataFile('your_package-default.txt') is None:
    return

So we must add a file with the name your_package-default.txt in profiles/default. The contents don't really matter; it can be something like this:

Flag file for the import handler of your.package

Proper catalog.xml

In our case we assume we still want the extra metadata in the catalog brains, so instead of the catalog.xml from the beginning we will have this one:

Happy indexing!

Setup your buildout

Soon: add Plone==4.0a1 to the eggs in your buildout.cfg

Now: add your egg to the development buildout: http://tr.im/TryPlone4

Example buildout.cfg for Products.Poi

[buildout]
extends = buildout.cfg
auto-checkout +=
    Products.Poi
    Products.AddRemoveWidget
    Products.DataGridField

[sources]
Products.Poi = svn https://svn.plone.org/svn/collective/Products.Poi/trunk
Products.AddRemoveWidget = svn https://svn.plone.org/svn/archetypes/Products.AddRemoveWidget/trunk
Products.DataGridField = svn https://svn.plone.org/svn/archetypes/Products.DataGridField/trunk

[instance]
eggs +=
    Products.Poi

Things to test

• Does bin/buildout work?
• Does the instance start up in the foreground?
• Can you install your product?
• Can you create/view your content type?
• Do the tests pass?

What to change

http://plone.org/documentation/migration

BACKWARDS_COMPATIBILITY.txt in Plone package:

http://tr.im/backwards

Highlights

• Python2.6, Zope 2.12, CMF 2.2
• No more global_defines.pt

Poi Example 1

ImportError: No module named Interface

• Remove any imports of the Interface package.
• Remove any __implements__ lines from your content types.
• http://dev.plone.org/collective/changeset/100276

Poi Example 2

ImportError: No module named GlobalTranslationService

• use zope.i18n

Old:

from Products.PageTemplates.GlobalTranslationService \
    import getGlobalTranslationService 
ts = getGlobalTranslationService() 
msg = ts.translate('Poi', msg, context=self.context)

New:

from zope.i18n import translate 
msg = translate(msg, 'Poi', context=self.request) 

• http://dev.plone.org/collective/changeset/100277

[Update: you need to pass the request instead of the context.]

Plone 3.x and 4?

Compatibility can work:

try:
    from beer import Grolsch
except ImportError:
    # BBB
    from beer import Heineken

or maybe not:

install_requires=[
    'Plone>=4.0dev',
],

Limi's work list:

• I am going to work on the sunburst theme, completing the css, integrate jquerytools, testing in all browsers.
• Figure out packaging strategy for Plone 5, decide on theme selection.

Limi's wish list:

• Our template customization story and jbot.
• Testing and updating add-on products for Plone 4. Key to the success of Plone 4.
• Next generation CacheFu: five.caching. CacheFu was built for Plone 2.1
• I want LinguaPlone to love Dexterity.
• Look into http://stackoverflow.com to answer questions instead of the plone-users list. Theme it with xdv. There are a lot of frustrated .net developers on stack overflow, so it is good if they see us.
• QA on collective.xdv (binary eggs on Mac).
• Reduce KSS dependency.
• Timezone support
• HttpOnly cookies in Plone 4
• Make all tests exercise unicode in Plone 4. Don't just add ascii as title when testing a portlet. Add umlauts, harts, etc.
• Scripts to build VMWare and Amazon and ISO images from the Unified Installer.
• Nightly performance tests. See if performance suddenly drops after a recent change. Talk to me if you want a list of things to check.
• Version ping daily or weekly to see what versions people are using. Opt-in.
• Typogrify filter. Python script to change -- into something prettier.
• Collections UI improvements. It is so much better in Plone 4. Rob and Geir have worked on this, so get pointers from them if you want to work on this.
• Better commenting system
• css improvements for plone.org and Trac
• Marketing:
  • Get a nice Plone 4 feature overview
  • Make comparisons of Plone versus other systems. Not one-sided things; guide people into using the right tool.
  • Move plone-announce to Feedburner. We have about 500 to 1000 addresses that have to be subscribed over from plone-announce to Feedburner.
• Clean out the issue tracker.
• Reproduce bugs, close irrelevant ones (like for NuPlone which we are not going to fix).

Topics by others

• Wolfgang: LinguaTools, helper set of rules which allow doing things to multiple translations of an object in one go, to avoid repetitive work. We want to make it nicer, more watertight, more modular, we want more tests.
• David: we find E-commerce difficult. We want to make PloneGetPaid easier, and want to move it to the collective.
• Andreas Zeidler: Plone 4 performance. Blob support, unified folders. Fix remaining bugs, update documentation about how to enable blob-related fields. If you have a large zodb with files, please donate it for testing.
• Tom: funkload automated performance testing.
• Tom: git integration for Plone core developers.
• Charlie Clark: core CMF, folder listing view based on formlib
• David Glick: Update add-on products to Plone 4; don't wait till Plone 4 is released. We can assist you now. Document common problems. Make list of products that are Plone 4 compatible.
• Otto, an http publisher without the overhead.
• Rob Gietema: Deco, Dexterity integration.
• Mike Holm: develop a better presence for education, build a website to give info to groups working in the education sector.
• Several people: make video content of first-class citizin. Plumi, p4a video, best of breed video problems, create a definitive buildout config. Plone 5 will have tiles for audio and video. Also: we have catering and coffee all day.
• Nate Aune: Banjo, point-and-click easy theming in Plone with Deliverance. Find me with the video sprinters.
• Dylan Jay: hostout, deploy Plone to any server you want with one command. I want Amazon ootb, figure out some best practices. Integrate with the ZopeSkel UI.
• Amberjack, tutorial tour product. We could help some help from someone who knows kupu.
• Ramon Navarro: multilingual, LinguaPlone for Dexterity, relations.
• Carsten Senger: content import and export for Plone. transmogrifier. How many options are really necessary, what could the UI be.
• Jens Klein: AGX modeling framework, python egg generation part, generate Dexterity types.
• Chris Calloway: ZopeSkel, had a BBQ sprint three weeks ago, we have som work left from that, package up the web interface, templates for Plone 4, Dexterity, WSGI, refactored into sub packages, documentation, fix mailing list.
• Christian Scholz, (MrTopf): social media.
• Bruno: Singing and Dancing
• Jordan Baker: roadrunner fast testing framework, not working so well with Plone 4 and Dexterity, we want to add tests, continuous integration server, Mac OS.
• PloneSocial, RPX (OpenId-like)

Meet up in the hallways, the open area where we gathered the whole week, and talk to the topic leaders.