Alessandro Pisa: Really Unconventional Migrations

Migration is just a matter of state, moving from state A to state B. The customer does not care how you get there, but you should, or it costs you too much time.

We migrated a site from Plone 4.1.3 to 5.0.6, and from a custom UI to Quaive, and from Archetypes to dexterity. The standard path is fine. But we also needed to upgrade Solr and several add-ons. And we had 10 GB of Data.fs and 300 GB of blobs, about 200K ATFiles, and previews, and versions, and about 5000 users.

For the standard path, getting the fresh data would take hours, upgrade of Plone hours, to dexterity days, and add-ons unknown. In one weekend we would need to block 5000 users. That was not really an option. So the target was less than one day.

Three environments: production on several machines, staging environment, and a migration server that was not very big.

Tip: use a migration view. You have well organised code, and have fast development with plone.reload, and you can curl it. This is better than having a script that you run with bin/zeoclient run. You get a well defined, reliable and clean upgrade path.

To rsync 300 GB of blobs takes time. Can we start a migration without blobs? Yes, by using experimental.gracefulblobsmissing. How to start:

• prepare migrating env
• copy Data.fs and Data.fs.index
• rsync blob in the background.

Some Plone upgrade bottlenecks:

1. The portal_catalog: if an index is added or needs reindexing, all objects need to be awoken and for us it took 45 minutes.
2. portal_catalog
3. portal_catalog...

Solution: go brainless. Clear the portal_catalog and run the standard Plone migration. This takes no time. For us this worked, because we wanted to touch all objects later anyway. But some upgrade steps might depend on having a fully filled catalog.

Add-ons. You may need newer versions. They may not yet be compatible. Solution: straight to the goal. We made a Plone 5.0.6 buildout with only the new code and packages that we wanted to use. So we had classes in the Data.fs that no longer existed. We used from plone.app.upgrade.utils import alias_module and used that for replacing missing classes with tiny classes that had just the few methods that were needed during migration. We cleaned up the persistent import and export steps from portal_setup before we started. We cleaned up the skins. So we focused on the result, and not on packages that would not be there anymore.

Biggest problem: @@atct_migrator for migrating from Archetypes to dexterity. This basically doubles your blobs because the old blobs are still there. It took too long for us. We needed to avoid recreating the blobs. In a migration step, for each object, we deleted it from the parent, set the newly wanted class, and added it to the parent again. Fields needed to be adapted or updated, like from DateTime to datetime. For each blob in a field, we created a new NamedBlobFile, and directly set _blob to the old blob data, so that no new blob had to be created on the file system. We never actually needed to read the blobs, so it was fine that the blobs were taking time to rsync.

Solr migration could be done in parallel, in half an hour. Then atomic updates in about two hours. SearchableText was not touched, which would have taken to long with all the parsing of text from PDF and Word.

In many cases, the standard Plone migration is fine. So if you can, just use it.

Note that we also needed to handle other catalogs, and link integrity, and some other stuff.

Philip: plone.app.contenttypes has a function for changing the class too. And functions for temporarily switching off linkintegrity.