https://ugent.be is a Plone 4.3 site of a large Belgian university. Started in 2002 as Zope/CMF site. 90.000 pages, sub sites, 40.000 students, hundreds of editors, 90 add-ons. They wanted to move to Plone 5.2 and Python 3.

• Traditional in-place migration: too manu add-ons, no one-on-one mapping possible.
• Transmogrifier: not yet Python 3 at the time, too much magic hidden in too many places with blueprints.
• So: custom migration solution.

Content types: standard types, plus four custom content types, including PloneFormGen. So that is quite reasonable. There is extensive usage of archetypes schema extenders.

Start: analyze and investigate your dependencies. - Based on Archetypes? Obsolete, replace. - No longer needed? Remove it. In Confluence we compiled a big table with for each package the basic information of how we would handle it: upgrade it, replace it, unknown yet, status of Python support.

Start with a minimal Plone 5.2 setup. Add one verified Python 3 compatible add-on at a time. Test extensively. Focus on content types first. Things like portlets can be handled later.

You need an export. We used a customized version of collective.jsonify. Core numbers: 90.000 json files, 55 GB data, 90 minutes, binary files base64 encoded.

We exported portlet assignments, default pages, layout information, workflow state, local roles, and pre-computed values for further efficient processing.

So we had 90.000 json files on the file system. We imported this in ArangoDB. Why use such a migration database? This allowed us to import only some portal types, or do parallel imports, and test complex migration steps like for PloneFormGen.

We briefly tried MongoDB, but that could not handle data over 16 MB. The json could be dumped unchanged in ArangoDB. This took 45 minutes.

Now we need to import this into Plone. Clean Python 3.7, Plone 5.2, plus the minimal set of packages needed for the content types. Import via plone.restapi. On top we have a dedicated migration package with special views. This handled things like translating between UIDs and paths.

The "magic" migration script is based on configuration in YAML.

• Phase 1: pre-check the migration, remove target site if it already exists from previous test, create new Plone Site, install add-ons.
• Phase 2: create all Folders, query ArangoDB for this.
• Phase 3: create all non folders.
• Phase 4: global actions. Check and migrate paths to UIDs in rich text fields. Assign portlets. Other specific fixup operations, like reindexing.

We migrated PloneFormGen (Archetypes) to easyform (dexterity). Export: one JSON for the FormFolder, one JSON file per field and action adapter. In easyform this needed to be turned into an EasyForm instance and a schema.

Topics (AT) to Collections (dexterity). Code was largely taken over from the plone.app.contenttypes migration.

From AT schema extenders to dexterity behaviors. First we made a list of which there were, are they in use, what do they do? Check which dexterity replacements there are. Create new behaviors.

Migrate packages to Python 3. This is mostly covered by talks of Philipp Bauer and David Glick. Common problems: utf-8 versus unicode, import fixes, implements to implementer. I rarely used the 2to3 and modernizr tools.

Some reimplementations: - portal skins to browser views - some packages with AT replaced with new packages with dexterity

Other problems:

• improper file and image metadata
• migration of vocabulary values, like old to new departments
• repetitive cycles: always a bug occurs after a day of migration right before the end.

Quality control: - you need to check that migrated content and configuration is complete - "works for me" is nice, but others need to check too

Most of the packages have been removed, the setup is much smaller.

Status: - content migration is complete - must be tested in detail - integrate with the new theme, test this - need a replacement of a specific membrane usage - need work on castle/cas plugin

Takeaways: - Export Plone to JSOJN: 2 hours. Fast. - Import JSON to ArangoDB: 45 minutes. Fast. - Import ArangoDB to plone.restapi: 36-48 hours. Painfully slow. - 1.5 - 2.0 seconds per content object on average - cannot parellellize this import, because you would get conflict errors - So Plone and ZODB and painfully slow for creating mass content.

Question: in a similar setup we did a live migration from a live site to a separate new site. Did you consider that?

Answer: I tried this for other sites, but here I wanted to be able to partial imports, independent of the live site.

Question: default migration patches away all kinds of expensive indexing. You might want to consider looking at that. Migrating ten items per second is possible, although that is inline migration. And can you share the code, especially for the easyform migration?

Answer: could be done, but is not the primary focus of the budget currently.

A slightly older version of the code is on community.plone.org.

Gatsby Preview gives you hot reloading of content. So you change something in Plone CMS, and it immediately is visible on your Gatsby site.

Plone CMS -> Web hook -> Gatsby source plugin. When content is created, edited, deleted, we will fire a websocket event.

When you change the title of a folder, you should update the breadcrumbs of the children too, and also navigation in several places. It took me a long time, but I have implemented this.

I have created a Gatsby theme for Plone: https://github.com/collective/gatsby-theme-plone

You can measure the performance of your website in the milliseconds it takes to handle a request, or how many requests per second. But this talk is about subjective speed: how fast does it feel? The contents of the page come to us at the speed of light. But it has to show up on the screen, and then your brain needs to process it.

And when you do an online payment, it may be irritating when it is slow, but even worse when you just end up on a homepage without indication of success or failure.

Perceived performance is dependent on the actual performance, the UX (user experience), and the expectec performance.

If your brain is active, time seems to move faster. If a page takes less than a second to load, do you really need a spinner or progress bar? The Polar app at some point changed to show a spinner for every action. Nothing changed in the speed, but reviews dropped: it reminded you that you had to wait all the time.

So instead, you may be able to distract the user. The user needs time to focus. So don't show a blank page, but show a skeleton page, which has roughly the structure of the page. Then the user already has a chance to see where the focus should be. I hope Plone will do this.

Show a spinner for ten seconds: slow. Show a simple animation for ten seconds: you don't notice that it is just as slow.

The perception of performance is often just as important as actual performance.

Collections are hard to explain in user trainings. In most cases you have to explain default pages first. Can we make this simpler? In Volto we do this by having folderish pages.

What if we kill all content types, and rely on one content type to rule them all? Confession: I actually voted against folderish pages in the Framework Team twice. So I was sceptical. With Volto we thought, let's give it a try, and we got overwhelmingly positive feedback. It avoids cognitive overhead.

Same with the Save button. Pastanaga puts this in the top left corner of the page, and I hated it. Then I actually tried it for 30 seconds and loved it.

Composite pages. The idea has been around for ages, and there have been lots of add-ons that do this, but it was never really good enough for core. Always a problem for migration.

Naming things. Some names are open for discussion. We renamed tiles to blocks. We renamed proxy to teaser. Topics, Smart Folders, Collections: in Volto we simply call this Listings.

I rediscovered a blog by Alexander Limi, one of the original creators of Plone: https://limi.net/things-plone Still an interesting list after all these years. And I recognize parts in the Pastanaga UI.

Plone in 2019 is tiny. This means we cannot afford to reinvent the wheel over and over again. So Volto builds on React, which has far more contributors. We tried Angular and Vue too, but React seems the best system. And we use the standard stack with React, Redux, Semantic UI. This will be very familiar to most React developers.

We have trained people with zero experience in html and css, and in three days they could basically theme a Plone Site, even though we only started talking about Plone and Volto in the last two hours of the last day. It was really amazing.

With Plone 6 we aim to simplify the stack. There are several PLIPs (PLone Improvement Proposals), like having the Plone Site root be a dexterity item. Most of these are optional, they would be nice to have for cleaning things up, but they are not strictly necessary. You can use "Plone 6" today:

• Plone 5.2
• Python 3
• Volto 4

Carrot and stick: selling a Plone upgrade just because you get Python 3, and not the unsupported Python 2 (stick), may not be enough to convince your customer. As extra carrot you can say: you can have Volto.

You can try Volto out live: https://volto.kitconcept.com (Still Volto 3 at this point.) This weekend, come join the code sprint at the conference.

The upgrade-step:directory ZCML directive allows us to use a new upgrade step definition syntax. Less room for typos. Upgrades in the given directory are auto-discovered. Each directory is a generic setup upgrade profile. You no longer need a profile version in metadata.xml.

With the bin/upgrade command you can upgrade your Plone site, install an add-on, upgrade an add-on. This includes progress logging, so you see that something happens during long upgrade steps.

Why do we write upgrades? We want to consistently alter content and configuration. Prevent undocumented changes through the Plone UI: your policy package should have the needed configuration, without relying on manual changes done on the live site. We want to apply a minimal set of changes, instead of reinstalling an add-on.

We have several helpers methods, for example for:

• class migration and in-place migration
• rebuild, add, remove indexes
• update object or workflow security

And there is much more, like deferrable upgrade steps, possible because we separately track which upgrade steps have been executed.

We are working on getting it running on Plone 5.2.

Code: https://github.com/4teamwork/ftw.upgrade