Andreas Jung: Migrating a large university site to Plone 5.2

published Oct 24, 2019

Talk by Andreas Jung at the Plone Conference 2019 in Ferrara.

https://ugent.be is a Plone 4.3 site of a large Belgian university. Started in 2002 as Zope/CMF site. 90.000 pages, sub sites, 40.000 students, hundreds of editors, 90 add-ons. They wanted to move to Plone 5.2 and Python 3.

Traditional in-place migration: too manu add-ons, no one-on-one mapping possible.
Transmogrifier: not yet Python 3 at the time, too much magic hidden in too many places with blueprints.
So: custom migration solution.

Content types: standard types, plus four custom content types, including PloneFormGen. So that is quite reasonable. There is extensive usage of archetypes schema extenders.

Start: analyze and investigate your dependencies. - Based on Archetypes? Obsolete, replace. - No longer needed? Remove it. In Confluence we compiled a big table with for each package the basic information of how we would handle it: upgrade it, replace it, unknown yet, status of Python support.

Start with a minimal Plone 5.2 setup. Add one verified Python 3 compatible add-on at a time. Test extensively. Focus on content types first. Things like portlets can be handled later.

You need an export. We used a customized version of collective.jsonify. Core numbers: 90.000 json files, 55 GB data, 90 minutes, binary files base64 encoded.

We exported portlet assignments, default pages, layout information, workflow state, local roles, and pre-computed values for further efficient processing.

So we had 90.000 json files on the file system. We imported this in ArangoDB. Why use such a migration database? This allowed us to import only some portal types, or do parallel imports, and test complex migration steps like for PloneFormGen.

We briefly tried MongoDB, but that could not handle data over 16 MB. The json could be dumped unchanged in ArangoDB. This took 45 minutes.

Now we need to import this into Plone. Clean Python 3.7, Plone 5.2, plus the minimal set of packages needed for the content types. Import via plone.restapi. On top we have a dedicated migration package with special views. This handled things like translating between UIDs and paths.

The "magic" migration script is based on configuration in YAML.

Phase 1: pre-check the migration, remove target site if it already exists from previous test, create new Plone Site, install add-ons.
Phase 2: create all Folders, query ArangoDB for this.
Phase 3: create all non folders.
Phase 4: global actions. Check and migrate paths to UIDs in rich text fields. Assign portlets. Other specific fixup operations, like reindexing.

We migrated PloneFormGen (Archetypes) to easyform (dexterity). Export: one JSON for the FormFolder, one JSON file per field and action adapter. In easyform this needed to be turned into an EasyForm instance and a schema.

Topics (AT) to Collections (dexterity). Code was largely taken over from the plone.app.contenttypes migration.

From AT schema extenders to dexterity behaviors. First we made a list of which there were, are they in use, what do they do? Check which dexterity replacements there are. Create new behaviors.

Migrate packages to Python 3. This is mostly covered by talks of Philipp Bauer and David Glick. Common problems: utf-8 versus unicode, import fixes, implements to implementer. I rarely used the 2to3 and modernizr tools.

Some reimplementations: - portal skins to browser views - some packages with AT replaced with new packages with dexterity