Enfold has been working on a secure cross platform mobile application the past eight months. Walk through of the Requirements, Security, Flutter framework, Backend configuration of Plone, Authentication and Lessons Learned. Our goal is to have a free public release of a limited version of the application Q4 2021.

It was an adventure for us. The core team never built a large mobile application. We did not know what we did not know.

The big picture:

• A mix of devices (Android, iOS) needs to synchronize files to and from Plone.
• All services are self-hosted in GovCloud. So we have no central database or server that we control.
• If this becomes a success, then future phases may require a lot of certification of the codebase, code reviews.
• Initially we worked with 15k devices, supporting 40k would be a success, the ceiling that we might support is 300k.
• Users are completely offline for longer times.

We used Flutter to create a React native app, see https://flutter.dev It is a UI toolkit. Why did we use Flutter?

• It is cross platform mobile.
• It uses Dart, which is statically typed, making code analysus much easier.
• Google seems to be prioritizing developer user experience, it really shows of quite a bit.

Dart has asynchronous code as a first class citizen. Quite different from Python. Runtime reflection (pdb) is unavailable. It has good ergonomics, with generics and closures. It is a driving force behind the Flutter toolkit.

Thoughts on mobile development:

• It is a lot to take in.
• You need to understand lots of languages, for us: kotlin/java, swift/obj-c, and Dart.
• No idea how to test platform integration.
• Native libraries are managed using Cocoapods/Gradle. Flutter drives those. Setting it up is yet another new thing to learn.
• There are lots of inconveniences, like how do you read sqlite off a device, because that is how we store some of the info?
• Also inconsistencies: if the app works on an emulator, that does not mean it works on a device.

On the server side:

• Plone operating as a Webdav server
• We need to support OIDC (mod_openids/oauth2)
• Not many writes, maybe 100-1000 per day, but lots of reads.
• 20k+ devices daily

Alternatives to the server/protocol could be nice:

• Honestly, are there any standards other than WebDav?
• An S3 api would be reasonable.
• So ideas are welcome, let me know.
• We have been working on prototypes with guillotina_webdav + openidc.

The good parts of Flutter:

• UI/UX development is very fast, with lots of widgets. We had two developers who were used to Angular, and they took it up quite fast.
• The bridge to native code (Pigeon) is straight forward.
• Drift is an amazing sqlite library.
• Riverpod for state management
• Dependency management is good (flutter pub). You can tell that they learned a ton from others, like pip. Except that very occasionally the package cache is broken so you need to clean it.
• They have a good community, with add-ons.

The not so good parts of Flutter:

• Inconsistent platform features, like WorkManager (Android) versus NSUrlSession (iOS)
• Dependency churn: often new versions come in, which you then need to check.

The mobile app:

• We are still wrapping up the remote file operations.
• After we deploy into production, we will improve the UI.

Yes, we hope to open source the synchronization framework, and maybe the foreground/background transferring subsystem. Yes, we have built a Flutter web-app of this, but it looks just like the mobile app currently. Needs a separate layout really. No, we have not done a Desktop app.

At last year's Plone conference, I presented Pyruvate, a WSGI server implemented in Rust (and Python). Since then, Pyruvate has served as the production WSGI server in a couple of projects. In this talk I will give a project status update and show how to use Pyruvate with Zope/Plone and other Python web applications and frameworks. I will also present some use cases along with benchmark results and performance comparisons.

WSGI is the Python webserver gateway interface. It is the default way for Python apps to talk with a webserver.

During the Python 3 migration of Zope and Plone, WSGI replaced ZServer as the default, since Plone 4. The ZODB is not thread-safe. This means there is a limited choice of WSGI servers. The Zope docs recommend only two: waitress and Bjoern. Other popular servers showed poor performance for Zope.

Rust is a new, popular programming language. It can also be used to extend Python packages. The cryptography package does this, and we all use this package.

Pyruvate from a user perspective:

pip install pyruvate

Then create an importable Python module:

import pyruvate

def application(environ, start_response):
    """WSGI application"""
    ...

pyruvate.serve(application, "0.0.0.0:7878", 3)

Using pyruvate with Zope/Plone. In buildout you add pyruvate to the eggs of your instance, and use a different wsgi-ini-template.

• Pyruvate supports active Python versions, currently 3.6-3.10.
• Using Mio to implement I/O event loop.
• Worker pool based on threadpool, 1:1 threading.
• A PasteDeploy entry point.
• It is stable since 1.1.0.
• Supports Linux and MacOS
• Hosted on Gitlab
• There are binary packages for Linux, so there you don't need a Rust compiler to install it.

Let's see about performance. As starting point: Performance analysis of WSGI servers by Omed Habib, see Appdynamics blog. This is a performance comparison of 6 WSGI servers, published in 2016. Tested were: Bjoern, CherryPy, Gunicorn, Meinheld, mod_wsgi and uWSGI.

Benchmarking tool was wrk. The test WSGI application simply returns some headers, and the text "OK".

I changed the original setup. Swapped Meinheld and mod_wsgi for Pyruvate and Waitress, swapped CherryPy for Cheroot. Use Python 3 only. Now we look at the metrics.

Number of requests served:

• Bjoern is by far the best.
• Pyruvate does not do so well with lower number of simultaneous connections, but with higher numbers, it jumps to second place after Bjoern.

CPU usage:

• Bjoern is single threaded, so is 100 percent, the others can do more.
• Gunicorn starts the best, but drops in performance with more simultaneous connections.
• Pyruvate starts slightly below it, but sustains it better.

Memory usage:

• More or less okay for all.
• Except uWSGI, where memory usage steadily goes up.

Errors:

• With increasing load, all servers show errors on higher load.
• Except waitress and Pyruvate.
• uWGI always shows errors for every single request, so something is wrong. But it still serves the request. So maybe an issue of the benchmarking tool.

Why is Bjoern so much faster?

• Good implementation, many optimizations.
• It is single threaded. Switching from single to multi-threaded comes with benefits and costs.
• Shared access to resources adds to complexity.
• Offloading requests to worker threads only makes sense when there is something to work on, which is not the case in this benchmark.
• Python's GIL generally makes multithreading less effective.

Let's look at benchmarking Plone 5.2.5. Testing with Bjoern, Pyruvate and Waitress. All three serve the Zope root about 600 requests per second, and this stays the same when simultaneous connections increase. Serving the Plone root: all three about 27 requests per second.

Number of errors, mostly socket errors:

• Bjoern starts giving errors a bit earlier, starting from ten simultaneous connections, but it holds up well.
• Pyruvate and waitress start giving errors at 50 simultaneous connections.
• With 100, Pyruvate does a lot worse than Bjoern, and waitress even more than that.
• So you really can't have that many simultaneous requests to Plone.

CPU usage is the same, all 100 percent. In this setup I have used one worker for each server.

Serving /Plone as json: Bjoern is slightly better than Pyruvate, which is slightly better than waitress, but all are around 500 requests per second, quite close to each other.

The most interesting is number of requests served when getting a 5.2 MB blob from blobstorage. Bjoern is around 200, Pyruvate 180, so close, and waitress a lot worse, dropping from about 80 to 40.

Next setup: 2 threads for Pyruvate and waitress. Pyruvate is then better than Bjoern. Waitress starts better, but cannot keep up.

Using 2 threads but 4 CPU, Both Pyruvate and waitress are a bit better than Bjoern, and keep it up.

Conclusions from benchmarking:

• Bjoern is the winner when using a single worker for all URLs except /Plone.
• When serving a more complex page such as /Plone, there is no real difference in the number of requests served, but Bjoern is showing errors a bit earlier.
• Adding one thread plus sufficient resources lets both Pyruvate and waitress perform better than Bjoern.
• All configurations failed to sustain higher loads, more than 50 connections.
• Bjoern and Pyruvate are serving blobs a lot faster than waitress.
• Pyruvate can challende waitress in all scenarios.
• When adding worker threads, Pyruvate seems to make better use of resources than waitress.

As test I setup Apache for fair balancing between two ZEO clients on Plone 5.2, one served by Pyruvate, one by waitress. Consistently more requests are sent to Pyruvate (53 percent).

Code: https://gitlab.com/tschorr/pyruvate

Problem: Volto has no equivalent of a viewlet. Solution: slots. They can be management slots, presentation slots, below-footer slots.

One reason: we try not to customize the main template.

Volto also does not have portlets. Well, if you really want them badly enough, you can have them. There is a PR in plone.restapi to export portlets, so you could render them in Volto.

Idea: reuse Volto blocks for layout.

Plone has had portlets for a long time, and it is very useful, especially for smaller sites. You should not have to be a web developer to change the website layout. Portlets give site administrators some power to influence the look of their own site. We should keep that possibility.

Volto's slot proposition:

• Simplify configuration. Portlets in Classic need too many files.
• Volto blocks are very expressive.
• Require Modify Portal Content permission for slots.
• UI Power: give more capabilities: - atomic blocking of parent blocks - override parent blocks

How can we use them?

• Sidebars: listings, info boxes, navigation
• section headers, content

Current status: big PRs on plone.restapi and Volto. Overall the basic functionality is 60-70 percent ready. I will do a live demo.

collective.exportimport is the latest addition to our migration toolbox and we have achieved good results with it with upgrading Plone sites to Plone 5.2. But A new 'side' use case with this add-on getting mature is distributing content trees between existing Plone sites. For example to create an online marketing campaign and deploying the setup to several country sites for translation and local use. I will demonstrate the 'content copy' use case, discuss current state and planned/wished improvements. As a related subtopic I will also touch on current capabilities and 'caveats' of exportimport when using this for migrations based on our current experience.

I will do exactly the same talk as Philip, but in my way and in half an hour.

As Philip said: "Migrations can be 'fun' again."

I started work on a migration in autumn of 2020. The Plone 4.3 site originally started in Plone 3, maybe even some left-overs from 2.5. I did all the usual stuff for inline migration to 5.2 on Python 3. And I found a banana peel. And another. And another. With such an old site, that has been migrated over and over, too many things can be lurking in the database that at some point bite you.

The main drawback of in-place migration: there are unknown unknowns. You never know how many dark things are still lurking in there until you have completely finished the migration. So: very hard to estimate.

You also require an 'intermediate' migration environment: you go from 4.3/Py2 to 5.2/Py2 to 5.2/Py3.

A lot of work has been done on in-place migration. It is stable. It works on standard Plone sites. But who really has a standard Plone site?

collective.exportimport uses ETL transformation. Transmogrifier uses the same idea/theory:

• you Export
• you Transform
• you Load

Actually, you often export, load, then transform/fix. So ELT/ELF. Benefits:

• No need for an intermediate Py2/3 environment.
• You don't touch your existing/old environment data.
• You only need to add collective.exportimport to your old site.

Now some technical tidbits.

There are several chicken-and-egg problems.

• You import a folder, this has a default page, but the page does not exist yet.
• Page A relates to Page B which does not exist yet.

So we import all content, and then import other stuff like default pages and relations afterwards.

Part 2 of this talk: Copying staging content.

Use case from a customer, Zeelandia. This customer has subsidiaries in several countries, also using Plone Sites. They wanted to create some content on one site and make it available on the other sites so local editors can adapt it to their language.

With content-tree support in exportimport, we can export a folder and import it in another site. This works! Except: this site has Mosaic, and we use persistent tiles.

Tile data is either stored:

• in a urlencoded part on the html field
• on the context of the item (plone.app.standardtiles)
• as a persistent mapping annotation on the context.

Problem: tile annotations are not yet exported. I asked my colleague Maurits: Can we fix this? Yes we can. Demo. We export the content tree to a shared directory that is accessible to all Plone Sites on the server. They use this now to export marketing compaigns.

We export the annotations using the dict_hook mechanism to check if the context has a Mosaic layout. We need some adapters/converters for richtext fields, and named files and images when they are in tiles. The context is not a content item, but a dictionary, that is why we need these extra converters. It would help to know the actual schema of the tile, and there should be ways for that, so we could improve this. Missing from tiles is: support for RelationValues.

Could we define a generic 'bundle' json Plone export format? Then we could export the content plus default pages plus relations, etcetera, in one json. And then we can import it in one go in the correct order.

Tips and tricks:

• Keep a log of what you do when, in which order, with number of items processed.
• collective.migrationhelpers has some fixers, which are being moved to exportimport.
• You can use collective.searchandreplace to fix most css class changes by using smart regexps.
• Check disk space, space in TMP.
• After import, do basic checks, like: can I edit content, especially rich text.