Daniel Jowett: Journeys with Transmogrifier & friends

published Oct 29, 2014, last modified Nov 16, 2014

Or: how not to get stuck in the Plone dark ages. Talk by Daniel Jowett at the Plone Conference 2014 in Bristol.

I have been doing software since 1997. Background in C, Java, SQL, etc. Using Plone since 2008. Came to Plone conference in Bristol in 2010 and everyone seemed to be cleverer than me. Four years later, I know that they are cleverer than me. But I have done four or five projects with transmogrifier and jsonmigrator.

Transmogrifier comes from a Calvin and Hobbes comic. Look it up. A cardboard box becomes a transmogrifier: it can turn you into anything you like: dinosaur, giant slug, whatever.

There are a few variants.

  • Plone transmogrifier uses two stages: export/import from/to Plone site via xml files or csv files.
  • Funnelweb: crawls and parses static sites for import, which are then pushed into Plone.
  • jsonmigrator: 1 stage process that crawls json views of an old site.
  • Blueprints also exist that can for example read from SQL databases.

We will focus on jsonmigrator.

When to use jsonmigrator

  • Good for migrating from old versions of Plone where you cannot install transmogrifier for exporting the contents.
  • Even from old Zope/CMF sites.
  • Particularly when you have no buildout.
  • When changing from archetypes to dexterity.
  • To clean cruft from an error-prone old site.
  • When scared to upgrade because you don't know what might bite you later. You don't know what you don't know...

Note: this is not the whole story: you still need to update your Plone version, add-ons, custom code, theme.

When NOT to use transmogrifier

  • From Plone 4.x.
  • Probably not from 3.3.6 where you already have buildout: just use the standard Plone upgrade path.

Technology stack

  • plone.app.transmogrifier + collective.transmogrifier as base
  • collective.jsonify
  • collective.jsonmigrator
  • extra pipelines:
    • transmogrify.dexterity
    • quintagroup.transmogrifier (I had some issues with it, but it has useful stuff, so you may need it.)

Setting up jsonmigrator

  • Duplicate your old Plone in your staging environment. Or use your public live site, but that is not recommended: it may have security issues, or you may run into corner cases where your site breaks. So do it on a copy please.
  • Install collective.jsonify there. If you don't have buildout, you may need to do it like this:
    • Download the egg to your Plone directory
    • Unzip it
    • Add it to your Python with setup.py install, which will also pull in simplejson. Use a virtualenv if possible.
  • Add some external methods in an Extensions directory, where you import the main jsonify functions. Add and use the methods in the ZMI of your old Plone Site. For example the get_item function, which you can then add to the end of a url to get the json version of content item. For a file you get the data included in base64 encoding.
  • Add collective.jsonmigrator to the eggs of your new Plone Site.
  • Go to the @@jsonmigrator view. Select a pipeline, update any settings, and run it.

In our case we had setup a really long pipeline to migrate from a Zope 2.6 site to Plone 4.

You can define your own pipeline sections, usually based on collective.transmogrifier.sections.manipulator to tweak data in the dictionary of the current item or .inserter to add data. Start with the collective.jsonmigrator.remotesource section for getting the data.

You may have to get your hands dirty when things don't exactly work the first time. Adding some print statements can already help to get the problem clear.

Caveats

  • It does not export/import users, though see collective.blueprint.usersandgroups.
  • Doesn't do portlets.
  • collective.jsonify is not security safe to use at a live site, as mentioned.

Credits

  • Transmogrifier: Martijn Pieters, Jarn.
  • Jsonmigrator: Rok Garbas
  • RCS: letting me loose on this
  • Calvin and Hobbes: Bill Watterson, building the first migrator.

Warning from Steve: I have run into problems with character sets. Paul: there are encoding and decoding blueprints that help, though it takes some getting.

Maarten: You are getting json data from an old site that is still running. Can't you save the json data on disk and use that? Erico: yes, you can, that is what I usually do. Maarten: I am using a blueprint for that, also because sometimes the export gave an error and sometimes the import.

Watch the video of this talk.

Keywords
plone ploneconf2014