Weblog

published Nov 03, 2021, last modified Nov 04, 2021

Fred van Dijk: 7 things that can surprise you when you start customising or developing for Plone

published Oct 28, 2021, last modified Oct 29, 2021

Talk by Fred van Dijk at the online Plone Conference 2021.

This is a basic rundown and summary of our beloved subjects like ZODB persistence. Traversal. The view/viewlet/portlet trinity. How is a call handled in Plone. The differences between zcml and generic setup. Utilities and the ZCA. restrictedTraverse. The Plone catalog.

These are surprises that I have encountered myself, or that I have seen on faces of people I have trained or worked with during the years.

  1. Everything can or could be done through the web (TTW).

Zope vision from the nineties. Why don't we use this dynamic language called Python so we can change things TTW? It's so easy.

  1. The learning curve.

It starts easy, but then you hit what we call a Z-shaped learning curve. Dynamic Python makes things easy, and then it makes things hard, at least when you put Zope on it, then CMF, then Plone. Plone the product on top of a CMS on top of a framework on top of a language. We have a product, a framework, a stack, so it is hard.

  1. Five levels of conceptual complexity.

It helps to teach all the levels. Give new users a drawer for each level, so they have a box that they can put some info in.

You have:

  • browser: html, css, js
  • frontend: transforms, templates, zpt, metal, diazo
  • application logic: views, viewlets, portlets, adapters, Zope Component Architecture
  • dynamism: GenericSetup, zcml, xml, zope schema
  • programming language: Python, buildout, pip
  • package WorkManager (OS): apt, rpm

4 Same language/formats on different levels:

  • XML is used for the ZCA, GS, zope.schema, Diazo rules
  • package manager: buildout, pip, setuptools, GS

But: there is no magic. It is just advanced technology.

Startup:

  • Python uses sys.path modules to start bin/instance
  • ZCA loads site.zcml, package includes, other zcml to change configuration.
  • Then we have a runtime environment with objects and classes, the ZODB. GenericSetup is then some XML that you can use to change the ZODB.

So the ZCA overrides components in runtime. The alternative is to edit core files, maybe compile them, and restart. Much less nice and not sustainable.

So now we have a Plone process running.

  1. Zope is not that complicated.

Over HTTP Zope gets a request, does traversal, finds an object in the ZODB, loads it in memory. Then on top of this object we show a browser view / template. The template hooks into a main template, maybe does some sub requests, some Diazo, and we end up with some html and we are done.

This is all still 'lies to children'. It is simplified until we are able to understand more. With increment exposure to these concepts, it will stick more. It is complicated, but there is no magic.

  1. Acquisition.

It is traversal in the wrong direction.

  1. When you try to explain things, you improve your own understanding.

There is so much Plone content online: training, docs, youtube, github, discourse. We all learn in different ways, with own preferences, reading, viewing. There are so many repositories on github that you can explore for new ideas. Just yesterday Philip did a talk about exportimport and afterwards I did it, but from a different angle. It helps.

The community is our biggest asset.

Annette Lewis and Will Gwin: From Zope to Plone: Thinking User-First During Migration

published Oct 28, 2021

Talk by Annette Lewis and Will Gwin at the online Plone Conference 2021.

Migrating a site is always a challenging task, but when you have dozens of subsites with specific brand standards and custom user functionality, the challenge becomes mammoth. Six Feet Up worked hand-in-hand with Purdue's College of Engineering to migrate their existing Zope site and its subsites into a new Plone instance running on Plone 5.2 / Python 3. Throughout the migration process, we considered the project scale, timelines, and limiting the impact on end users, all while managing the balance between user needs and best practices. During this presentation, you will learn:

  • why it matters to think user-first during migration,
  • about creative solutions for translating content and functionality into Plone, and
  • how to successfully migrate subsites.

An overview of the project:

  • HigherEducation always seems to go a bit slower, certainly with migrations.
  • Our previous CMS was built in Zope and was getting extremely old.
  • We have been using Python since 2001.
  • There were security concerns and modernization issues.

The impact:

  • Only 15+ content editors.
  • 40+ public facing subsites
  • 30,000 total pages.
  • 20+ departments and units

Why did we select Plone as the next CMS?

  • It is a modern CMS solution
  • Python-based, so that fit what we currently have.
  • It is built on top of the Zope web framework that we were already using.
  • We looked at Drupal, Wordpress, and more, but that would have been a too big undertaking.

This was in 2018.

Laying out the solution. From requirements to action. Who are our users and what do they want? Not just our direct client (Will and his team) but their clients/users.

Challenges during development:

  • Purdue University changed its brand look. We had to seamlessly blend subsites into the existing parent site.
  • Convert all content types and templates from Zope to Plone.
  • Keep sight of the user experience in both environments. Could they use the new environment without too much training, or needing to have too much tech knowledge?

Determine the project essence. Distill the requirements down into broad categories: accessibility, usability, flexibility, security.

What is the path to successful collaboration?

  • The absolute best might not be the right answer.
  • It's okay to say no to an idea, but you dhouls have an alternative ready.
  • Aim for the best, avoid the dangerous, end up somewhere in the middle sometimes.

On to our development goals:

  • Do things in a Plone way. Plone uses Zope, but Zope may do some things in a different way than is the best way in Plone.
  • So observe best practices.
  • Make it intuitive and keep it familiar for the editors.

Solutions at a glance

Migrating site content:

  • We wanted to move subsites one by one, as needed.
  • Translate existin content to Plone content types
  • We could re-import content over existing content non-destructively.

Theming: retrofitting Plone into an existing theme. That is what Diazo was made for, bridging the gap between Plone and the theme, especially since the theme is 'living', with subtle changes coming in often.

Each subsite had a browser view named local.css to change some things. Not really what you want in Plone, but they really needed it, as a way to make subsites or sections look different. So we added an action to edit the local css, inheriting from parent folders.

We created a subsite settings control panel. We used lineage.registry for this. All kinds of customizations can be done based on that, for example add extra text and links in the navigation menu. They used to be able to do this in Zope as well, but that was with various properties, and much more code oriented.

We use Mosaic for flexibility of layout. In Zope we had blocks for layout. Mosaic took this a step further into a nice UI with drag and drop. It gives faster site prototyping and development.

The sandbox: we wanted to have safe spaces for content experimentation. Completely separate from production environment. It is used for new user training and testing. It is quick and easy to reset.

This migration project has been a constant collaboration between the Purdue communications office, Engineering Computer Network, and Plone company Six Feet Up. The content editors feel empowered to make complex changes, without constant oversight from my team.

Alan Runyan: Building a Secure Cross Platform Mobile/Tablet Application (Flutter) using Plone as Backend Server

published Oct 28, 2021

Talk by Alan Runyan at the online Plone Conference 2021.

Enfold has been working on a secure cross platform mobile application the past eight months. Walk through of the Requirements, Security, Flutter framework, Backend configuration of Plone, Authentication and Lessons Learned. Our goal is to have a free public release of a limited version of the application Q4 2021.

It was an adventure for us. The core team never built a large mobile application. We did not know what we did not know.

The big picture:

  • A mix of devices (Android, iOS) needs to synchronize files to and from Plone.
  • All services are self-hosted in GovCloud. So we have no central database or server that we control.
  • If this becomes a success, then future phases may require a lot of certification of the codebase, code reviews.
  • Initially we worked with 15k devices, supporting 40k would be a success, the ceiling that we might support is 300k.
  • Users are completely offline for longer times.

We used Flutter to create a React native app, see https://flutter.dev It is a UI toolkit. Why did we use Flutter?

  • It is cross platform mobile.
  • It uses Dart, which is statically typed, making code analysus much easier.
  • Google seems to be prioritizing developer user experience, it really shows of quite a bit.

Dart has asynchronous code as a first class citizen. Quite different from Python. Runtime reflection (pdb) is unavailable. It has good ergonomics, with generics and closures. It is a driving force behind the Flutter toolkit.

Thoughts on mobile development:

  • It is a lot to take in.
  • You need to understand lots of languages, for us: kotlin/java, swift/obj-c, and Dart.
  • No idea how to test platform integration.
  • Native libraries are managed using Cocoapods/Gradle. Flutter drives those. Setting it up is yet another new thing to learn.
  • There are lots of inconveniences, like how do you read sqlite off a device, because that is how we store some of the info?
  • Also inconsistencies: if the app works on an emulator, that does not mean it works on a device.

On the server side:

  • Plone operating as a Webdav server
  • We need to support OIDC (mod_openids/oauth2)
  • Not many writes, maybe 100-1000 per day, but lots of reads.
  • 20k+ devices daily

Alternatives to the server/protocol could be nice:

  • Honestly, are there any standards other than WebDav?
  • An S3 api would be reasonable.
  • So ideas are welcome, let me know.
  • We have been working on prototypes with guillotina_webdav + openidc.

The good parts of Flutter:

  • UI/UX development is very fast, with lots of widgets. We had two developers who were used to Angular, and they took it up quite fast.
  • The bridge to native code (Pigeon) is straight forward.
  • Drift is an amazing sqlite library.
  • Riverpod for state management
  • Dependency management is good (flutter pub). You can tell that they learned a ton from others, like pip. Except that very occasionally the package cache is broken so you need to clean it.
  • They have a good community, with add-ons.

The not so good parts of Flutter:

  • Inconsistent platform features, like WorkManager (Android) versus NSUrlSession (iOS)
  • Dependency churn: often new versions come in, which you then need to check.

The mobile app:

  • We are still wrapping up the remote file operations.
  • After we deploy into production, we will improve the UI.

Yes, we hope to open source the synchronization framework, and maybe the foreground/background transferring subsystem. Yes, we have built a Flutter web-app of this, but it looks just like the mobile app currently. Needs a separate layout really. No, we have not done a Desktop app.

Lightning talks Wednesday

published Oct 27, 2021

Lightning talks on Wednesday at the online Plone Conference 2021.

Peter Holzer: My first Plone 6 site

This was in April 2020. I friend wanted a shop for his post cards. Lots of search filters.

We create a new Plone theme on Bootstrap 5. Updated our bda.shop stack.

See https://www.fotoeigenart.ch

Michael McFadden: Web developer confessions

I had a web form up where people could submit confessions, totally anonymous.

  • I used Perl
  • I sometimes test things in production.
  • I voted against removing the tag from HTML.
  • I store user passwords, as we often need to login as a user to fix their... stuff. We take precautions, but still.
  • I allowed editors to add css at a top level in the site.
  • I have a marquee tag on our site.
  • I am using iframes to include those who are unwilling to move into our big Plone Site. It is a way to do imports.
  • I use pdb on production all the time with a client instance.
  • I use
    for my page layout.
  • I don't setup the whole translation machinery on our Plone Site. Hello USA!
  • I use heavily duplicated code. What's wrong with coding with copy and paste.
  • I did not backup my data, and I knew the RAID aray was bad.
  • I think downtime is good for mental health.
  • I still hate javascript. And now we have node.
  • I don't always do web development. But when I do, I do it through Zope using plone_custom.css on a production site.
  • I fixed it with css. .class {display: none !important}
  • I suggested to a user to try it in another browser, like Edge. I am still ashamed.
  • I am not trained in UX or web design, but I do it anyway, because I just know what's right.

Fulvio Casali: Plone sponsorships

We are looking for financial sponsorship on a regular basis. Open source is a labor of love, but we also need money. The Plone Foundation supports the community financially for sprints, conferences, releases. We have administrative costs, like for trademarks, lawyers, hosting.

So you support independent media producers via Patreon, Substack, Medium, Youtube? How much? If not, maybe your boss does? Have you watched listened to the Plone podcasts?

Sponsors can be providers, customers, universities, companies that somehow use Plone. We couldn't do it without you.

Mail sponsor@plone.org.

Dylan Jay: Why Drupal won down under

Two stories of government websites, in this case UK and Australia, where other CMSes won the day. Maybe with Plone 6 we can gain some ground back.

PreviousNext is a Drupal shop in Australia. In 2012 aGov launched, a distribution of Drupal specifically for Australian Government websites. That was cleverly done. They organized a Drupal conf in Canberra, the capitol. Most of the stuff could not have been done without support by Acquia, the behemoth company behind Drupal. It helps to be able to say that they have got you backed.

Then the UK. In 2012 Gov.uk launched. 2013 Service manual published.

Ideas for Plone:

  • Open CMS for a specific government.
  • Acquia equivalent
  • How to sell o governments that have been burnt by Plone?
  • Target internal React developers.
  • Use a better term than 'enterprise'. Governance CMS for React Developers?
  • Concentrate on services, not websites?

Maik Derstappen: Current state of Mockup

We are moving Mockup and Patternslib to ES6. We replaced some Mockup patterns with patterns from Patternslib. Most mockup patterns are finished, just a few left that need a bit of work. Plone Classic UI frontend works basically, some issues left in control panels mostly.

Things will get easier. No more RequireJS yelling at you. Add-ons can provide, require and ship any javascript module. Plone will only load everything once, thanks to Webpack module federation.

Timeline: ES6 branch will be merged in the coming months.

Give it a try using the buildout.coredev repository and:

./bin/buildout -c plips/plip-3211-mockup-redone.cfg

Help us! Join us during the conference sprint or every Thursday on Discord, the classic-ui voice channel.

Johannes Raggam: Javascript integration

This is about Webpack module federation.

Goals:

  • add-on without recompiling
  • No or negligable code duplication, small bundle size

Hard to achieve with Plone 5 and RequireJS.

New concept: Webpack module federation, or actually just module federation, as it is not tied to webpack.

  • separate bundles
  • dependencies between each other
  • define shared bundles
  • define exported and remote modules
  • fallback to own dependency

David Glick: Snowfakery

Snowfakery is a tool for generating fake data that has relations between tables. Every row is faked data, but also unique and random, like a snowflake. To tell Snowfakery what data you want to generate, you need to write a Recipe file in YAML.

You may want realistic data in tests, but not production data.

I write a yaml file and tell it how many folders and documents to create. I let snowfakery export this to json. Actually, I have defined a slightly different format for Plone. This outputs a json file that can be used by collective.exportimport.

Snowfakery documentation: https://snowfakery.readthedocs.io/en/docs/

Philip Bauer: One small VS Code trick

For the training this year for the first time I used one editor for both Volto and Classic. This is because of one VS Code trick.

I use the Pylance language server for VS Code, which is fine, but it cannot initially find the Plone code. Simple fix for that: I have a script zopepy in my bin folder. Find it, copy the path, in VS code configuration go to 'Python: Select interpreter', and paste the path. After a few seconds, or maybe reloading an open file, it works, and VS Code finds all packages and modules.

Erico - New Docker images

Plone loves containers. Some have had a love/hate relationship with Docker, but we are over it. Maybe Docker is over it too.

We have a new generation of Plone Docker images:

- ``plone/plone-backend`` 5.2.6 and 6.0.0a1
- ``plone/plone-frontend`` Volto 14.0.0-alpha
- ``plone/plone-zeo`` is still available (5.2)
- ``plone/plone-haproxy``

Use your own Plone image, with an example extra add-on:

FROM plone/plone-backend:6.0.0a1
RUN ./bin/pip install "pas.plugins.authomatic --use-deprecated legacy-resolver"

The use-deprecated option should hopefully not be needed in the near future. Maurits has opened a [PR in pip](https://github.com/pypa/pip/pull/10574) for that.

All these images are based on pip. One point: do not use autoIncludeDependencies in your zcml, as this does not work with pip.

Examples for Docker compose: https://github.com/plone/plone-frontend#using-docker-compose

Thank you to the installers team: Jens, Alin, Silvio, Steve, Kim, Erico.

Thomas Schorr: Pyruvate WSGI Server Status Update

published Oct 27, 2021

Talk by Thomas Schorr at the online Plone Conference 2021.

At last year's Plone conference, I presented Pyruvate, a WSGI server implemented in Rust (and Python). Since then, Pyruvate has served as the production WSGI server in a couple of projects. In this talk I will give a project status update and show how to use Pyruvate with Zope/Plone and other Python web applications and frameworks. I will also present some use cases along with benchmark results and performance comparisons.

WSGI is the Python webserver gateway interface. It is the default way for Python apps to talk with a webserver.

During the Python 3 migration of Zope and Plone, WSGI replaced ZServer as the default, since Plone 4. The ZODB is not thread-safe. This means there is a limited choice of WSGI servers. The Zope docs recommend only two: waitress and Bjoern. Other popular servers showed poor performance for Zope.

Rust is a new, popular programming language. It can also be used to extend Python packages. The cryptography package does this, and we all use this package.

Pyruvate from a user perspective:

pip install pyruvate

Then create an importable Python module:

import pyruvate

def application(environ, start_response):
    """WSGI application"""
    ...

pyruvate.serve(application, "0.0.0.0:7878", 3)

Using pyruvate with Zope/Plone. In buildout you add pyruvate to the eggs of your instance, and use a different wsgi-ini-template.

  • Pyruvate supports active Python versions, currently 3.6-3.10.
  • Using Mio to implement I/O event loop.
  • Worker pool based on threadpool, 1:1 threading.
  • A PasteDeploy entry point.
  • It is stable since 1.1.0.
  • Supports Linux and MacOS
  • Hosted on Gitlab
  • There are binary packages for Linux, so there you don't need a Rust compiler to install it.

Let's see about performance. As starting point: Performance analysis of WSGI servers by Omed Habib, see Appdynamics blog. This is a performance comparison of 6 WSGI servers, published in 2016. Tested were: Bjoern, CherryPy, Gunicorn, Meinheld, mod_wsgi and uWSGI.

Benchmarking tool was wrk. The test WSGI application simply returns some headers, and the text "OK".

I changed the original setup. Swapped Meinheld and mod_wsgi for Pyruvate and Waitress, swapped CherryPy for Cheroot. Use Python 3 only. Now we look at the metrics.

Number of requests served:

  • Bjoern is by far the best.
  • Pyruvate does not do so well with lower number of simultaneous connections, but with higher numbers, it jumps to second place after Bjoern.

CPU usage:

  • Bjoern is single threaded, so is 100 percent, the others can do more.
  • Gunicorn starts the best, but drops in performance with more simultaneous connections.
  • Pyruvate starts slightly below it, but sustains it better.

Memory usage:

  • More or less okay for all.
  • Except uWSGI, where memory usage steadily goes up.

Errors:

  • With increasing load, all servers show errors on higher load.
  • Except waitress and Pyruvate.
  • uWGI always shows errors for every single request, so something is wrong. But it still serves the request. So maybe an issue of the benchmarking tool.

Why is Bjoern so much faster?

  • Good implementation, many optimizations.
  • It is single threaded. Switching from single to multi-threaded comes with benefits and costs.
  • Shared access to resources adds to complexity.
  • Offloading requests to worker threads only makes sense when there is something to work on, which is not the case in this benchmark.
  • Python's GIL generally makes multithreading less effective.

Let's look at benchmarking Plone 5.2.5. Testing with Bjoern, Pyruvate and Waitress. All three serve the Zope root about 600 requests per second, and this stays the same when simultaneous connections increase. Serving the Plone root: all three about 27 requests per second.

Number of errors, mostly socket errors:

  • Bjoern starts giving errors a bit earlier, starting from ten simultaneous connections, but it holds up well.
  • Pyruvate and waitress start giving errors at 50 simultaneous connections.
  • With 100, Pyruvate does a lot worse than Bjoern, and waitress even more than that.
  • So you really can't have that many simultaneous requests to Plone.

CPU usage is the same, all 100 percent. In this setup I have used one worker for each server.

Serving /Plone as json: Bjoern is slightly better than Pyruvate, which is slightly better than waitress, but all are around 500 requests per second, quite close to each other.

The most interesting is number of requests served when getting a 5.2 MB blob from blobstorage. Bjoern is around 200, Pyruvate 180, so close, and waitress a lot worse, dropping from about 80 to 40.

Next setup: 2 threads for Pyruvate and waitress. Pyruvate is then better than Bjoern. Waitress starts better, but cannot keep up.

Using 2 threads but 4 CPU, Both Pyruvate and waitress are a bit better than Bjoern, and keep it up.

Conclusions from benchmarking:

  • Bjoern is the winner when using a single worker for all URLs except /Plone.
  • When serving a more complex page such as /Plone, there is no real difference in the number of requests served, but Bjoern is showing errors a bit earlier.
  • Adding one thread plus sufficient resources lets both Pyruvate and waitress perform better than Bjoern.
  • All configurations failed to sustain higher loads, more than 50 connections.
  • Bjoern and Pyruvate are serving blobs a lot faster than waitress.
  • Pyruvate can challende waitress in all scenarios.
  • When adding worker threads, Pyruvate seems to make better use of resources than waitress.

As test I setup Apache for fair balancing between two ZEO clients on Plone 5.2, one served by Pyruvate, one by waitress. Consistently more requests are sent to Pyruvate (53 percent).

Code: https://gitlab.com/tschorr/pyruvate