Fred van Dijk: collective.exportimport: tips, tricks and deploying staging content

collective.exportimport is the latest addition to our migration toolbox and we have achieved good results with it with upgrading Plone sites to Plone 5.2. But A new 'side' use case with this add-on getting mature is distributing content trees between existing Plone sites. For example to create an online marketing campaign and deploying the setup to several country sites for translation and local use. I will demonstrate the 'content copy' use case, discuss current state and planned/wished improvements. As a related subtopic I will also touch on current capabilities and 'caveats' of exportimport when using this for migrations based on our current experience.

I will do exactly the same talk as Philip, but in my way and in half an hour.

As Philip said: "Migrations can be 'fun' again."

I started work on a migration in autumn of 2020. The Plone 4.3 site originally started in Plone 3, maybe even some left-overs from 2.5. I did all the usual stuff for inline migration to 5.2 on Python 3. And I found a banana peel. And another. And another. With such an old site, that has been migrated over and over, too many things can be lurking in the database that at some point bite you.

The main drawback of in-place migration: there are unknown unknowns. You never know how many dark things are still lurking in there until you have completely finished the migration. So: very hard to estimate.

You also require an 'intermediate' migration environment: you go from 4.3/Py2 to 5.2/Py2 to 5.2/Py3.

A lot of work has been done on in-place migration. It is stable. It works on standard Plone sites. But who really has a standard Plone site?

collective.exportimport uses ETL transformation. Transmogrifier uses the same idea/theory:

• you Export
• you Transform
• you Load

Actually, you often export, load, then transform/fix. So ELT/ELF. Benefits:

• No need for an intermediate Py2/3 environment.
• You don't touch your existing/old environment data.
• You only need to add collective.exportimport to your old site.

Now some technical tidbits.

There are several chicken-and-egg problems.

• You import a folder, this has a default page, but the page does not exist yet.
• Page A relates to Page B which does not exist yet.

So we import all content, and then import other stuff like default pages and relations afterwards.

Part 2 of this talk: Copying staging content.

Use case from a customer, Zeelandia. This customer has subsidiaries in several countries, also using Plone Sites. They wanted to create some content on one site and make it available on the other sites so local editors can adapt it to their language.

With content-tree support in exportimport, we can export a folder and import it in another site. This works! Except: this site has Mosaic, and we use persistent tiles.

Tile data is either stored:

• in a urlencoded part on the html field
• on the context of the item (plone.app.standardtiles)
• as a persistent mapping annotation on the context.

Problem: tile annotations are not yet exported. I asked my colleague Maurits: Can we fix this? Yes we can. Demo. We export the content tree to a shared directory that is accessible to all Plone Sites on the server. They use this now to export marketing compaigns.

We export the annotations using the dict_hook mechanism to check if the context has a Mosaic layout. We need some adapters/converters for richtext fields, and named files and images when they are in tiles. The context is not a content item, but a dictionary, that is why we need these extra converters. It would help to know the actual schema of the tile, and there should be ways for that, so we could improve this. Missing from tiles is: support for RelationValues.

Could we define a generic 'bundle' json Plone export format? Then we could export the content plus default pages plus relations, etcetera, in one json. And then we can import it in one go in the correct order.

Tips and tricks:

• Keep a log of what you do when, in which order, with number of items processed.
• collective.migrationhelpers has some fixers, which are being moved to exportimport.
• You can use collective.searchandreplace to fix most css class changes by using smart regexps.
• Check disk space, space in TMP.
• After import, do basic checks, like: can I edit content, especially rich text.