pip is the tool almost every Pythonista learns early to use. Plone 6 installs just with "pip install -c ... Plone".

But it needs more: Zope configuration, including existing or development of own add-ons, using newer package-versions, configuring RelStorage, CI-CD integration, building Docker images, deployment on Swarm or K8s, an ingress, some load balancing, caching, ...

The talk is not a tutorial but gives a 3000 foot view and acts as a starter to dig deeper.

In the past we only had Plone as backend. Now also the frontend, running on a different port, different process. The backend talks to the database.

The Plone frontend is a Node project. It pre-renders pages on first request, and for this it talks to the backend.

Plone backend is a WSGI server based on Zope, with a complete CMS Rest API. The backend still has all the Classic UI features in it, even when this is not used with the new default front end.

You would use pip to install the backend: pip install Plone. But it is not perfect, so I created mxdev to override constraints. Previously Buildout was used to generate a directory structure and configuration files, but currently you use cookiecutter-zope-instance for this. And then there is a WSGI server to start Zope, by default this is waitress.

A web request comes in at the web server (nginx, traefik), and then most requests go to frontend, api requests to backend, and the frontend talks to the backend for the first request. You can scale the backend horizontally by adding more backends (zeoclients). The backends talk to the ZEO database, with usually a shared filesystem for the blobs (binary large files). You could scale the frontend as well, if that turns out to be the bottle neck.

If you use multiple backends or frontends, you will need to put a load balancer in between.

If your sysadmins don't like this ZEO setup, you can use a relational database. This uses the RelStorage package. Postgres would be the best choice, but MySQL should work as well. The blobs are stored as large objects in the database, which is more performant for a transaction.

At some point, you need some kind of caching. You can go to a cloud provider for a cache. But you can do it yourself: use a varnish cache. You put this between the web server and backend, and/or between web server and frontend. You need to configure it. Under very high load you could even say: cache this request for three seconds.

Now on to hosting. Let's focus on Docker.

Disclaimer: Docker swarm could die in the long term. But it is nice to start with, and it gives you knowledge that is also useful for Kubernetes.

You get images from docker hub, or other container registries. Usually you will build your own frontend, so you need to create a custom image for this. For the backend you can also do this. You could create an image on your laptop and upload it, but it is better to build and store them in CI/CD, to avoid big uploads on slow connections.

Now it is time to deploy it. Basics: There is docker compose, but you should use Docker Swarm with 1 to n nodes, or same with Kubernetes. You need storage for your database. This could be a managed database by your provider. And you need fixed IPs.

Simple deployment for one site would be a single node docker swarm, 1 docker stack with all included, all in one yaml file. 1 traffic, 2 frontends, 2 backends, 1 postgres. The 2 frontends and backends are so that you can upgrade 1, bring it up, the upgrade the second and bring it up, so no downtime.

You can add varnish in here. Traefik sends all requests there. Varnish returns a cached response, or it adds a header. With the header, Traefik sends the requests to frontend or backend.

Now some tools. With Traefik UI you can inspect what is configured or wrong. Traefik has Let's Encrypt built in. Portainer is a UI and management interface for Docker Swarm and Kubernetes, and you can add this with an image. You inspect the state of the cluster, stop and start services, view logs, open a web console.

You really need a CI/CD system, otherwise this get nasty with images. You need workflows and a container registry. GitLab and GitHub have both.

Want to play with this? See the deployment training.

"Untested code is broken code" according to Martin Aspen and Philipp von Weitershausen. But tested code is broken code too: even if all tests pass, it is not a guarantee that everything works, even with 100 percent test coverage.

So why do we write tests?

• Avoid regressions
• Improve your confidence when you change the system, for example from Python 2 to Python 3.
• Really: to minimise the chance of being ashamed.

History

When did the first tests get into Plone? I got to October 7 2002. A kind of continuous integration was done first in 2004. Golden age of Plone testing was between say 2009 and 2013, when PloneTestCase was there, plone.app.testing was created as new version, there was a Testing Team, Jenkins was added in 2010. Travis CI was used since 2012, a lot of add-ons started using that. We got robotframework around 2012, for end user testing in the browser, using Selenium. In 2013 mr.roboto was born so we could make sure that when one package changed, the full test suite of all Plone was run.

In 2018 GitHub Actions started. In 2019 Cypress could be used as alternative for robotframework, Volto is using this. Around now, some tools around code analysis have gotten updated.

Our current process

• You make a pull request.
• GitHub Actions checks are run, mostly code analysis, in some cases building docs.
• With a comment on GitHub you can start Jenkins jobs.
• When everything is green, the PR can be merged

Future

These are some ideas, I will be asking questions.

• Are we using the right tools? Most of the Python community uses pytest, we use zope.testrunner. You can actually use gocept.pytestlayer now so your Plone testing layers are treated as pytest layers. You need to move the Python code into src. And don't mix doctests and unit tests in the same module.
• Browser testing: Selenium? Cypress? Playwright?
• Move from Jenkins to GitHub Actions? Thank you Gil Forcada for keeping Jenkins up and running so far!
• Can the tests go faster? Any easy wins? With parallelisation on my M1 I ran the tests in four minutes.
• Go to a monorepo? For things that are used outside of Plone: no. But it may make a lot of things easier. Recently I changed the minimum password length and needed changes in eight packages or so; with a mono repo this would have been much easier.
• Do Classic UI and Volto test the same features? Could the plone.restapi tests run on Guillotina or Nick?
• Which version are we testing? We are not testing the backend of Plone against Volto until it is too late.
• Do we have the right tests? 350 acceptance tests (robot plus cypress), 800 integration tests (Python), 8000 unit tests (Python plus Jest)
• What else could we be testing? Accessibility testing. Visual regression testing. Performance tests. Type checking. In VS Code: go to Python, Analysis, Type Checking Mode, and set it to basic.
• Error tracking
• Distributed tracing for seeing how long things take to execute.

Audience:

• The Classic UI tests could be split from the REST api tests.
• For pytest, you could move the tests out of the package code to the root.
• Could we make pytest the default for add-ons?
• Mono repo does not solve everything. We need to think about architecture. But fewer repos is good.

Plone 6.0 will be matched with the Volto version 16, which is an LTS version. Volto 16 is in alpha stage, a few more features are pending, hopefully merged during the conference sprint.

Volto 17, 18, 19 will be worked on, first as "canary" (testing) and then stable, and released faster. Volto 16 will still be supported then. Bug fixes and features will be back ported to Volto 16, but no breaking changes.

At some point Plone 6.1 will be released. We will have to see which Volto version goes with that, for example 18. It will be clear at the time.

Core features in Volto 16:

• Configuration registry
• Shadowing and the component registry
• Add-on architecture
• Seamless mode and internal proxy, with variables that can be used when building or when running.
• Routing
• Data fetching
• Server side rendering
• Theming
• Pastanaga editor

We should continuously curate these features. They should remain high quality, also when the React ecosystem constantly evolves.

What's ahead?

• React 18 is out. New Suspense SSR Architecture.
• Routing: currently we are stuck with RR5, React Routing 5. We have AsyncConnect and connected-react-router. We cannot upgrade to RR6, so not to React 18. Routing libraries have improved a lot lately, including data loaders when the route changes. Some modern frameworks for this: Remix and NextJS.
• Server side rendering: we need to modernise and simplify this. AsyncConnect is obsolete. New Suspense SSR is amazing.
• Global state handling: paradigm of global state handling and Redux has changed.
• Data fetching: this is again influenced by the Suspense SSR in React 18. Need to adopt to @tanstack/react-query.
• Reusable components: we can extract basic Volto components to their own add-ons, unconnected. Make it easier to use other JS frameworks with Plone 6 as backend.
• Build modernisation: go to yarn 3, pnpm. Use next generation builders/transpilers, like Vite, ESBuild, SWC? Integrate it in Razzle or in another boiler plate.

What features are coming next?

• New theming architecture. Would be nice to support Bootstrap. We would need to separate the theme and CMSUI, and use a vanilla, agnostic CMSUI based on neutral frameworks.
• Quanta: the new design system by Albert Casado. We need to make that happen, probably in Plone 7, see when this is ready. Use the Quanta toolbar. Implement the design system and update the story book.
• Slots: a mix between viewlets and portlets.Inheritable, lightweight, containing blocks. Assigned statically and/or defined in the backend.

More features, working on this during the sprint would be good, especially the first three:

• Row / grid and columns / group block
• Nested drag & drop blocks support
• Teaser block
• Relations control panel
• Improve transform run detection based on agnostic markers, JSON-LD like
• Subscribe-able context for every block (widgets)
• JSON-schema form generatio, adopt a specialised library.
• Rework the object browser, improve its api and features

Work on alternative backends: Guillotina, Nick.js.

Mining/extraction of mineral resources (for ICT and other usages) logically involves geological, technological and engineering concerns. However, it also deals with numerous other essential aspects, although less discussed by medias/citizen/experts, such as economy (“circular economy”), sociology (“social acceptability” or “perception of mining by the citizen”), ethics (“artisanal mining”), geopolitics (“strategic/critical commodities”, “national strategies”), environment (“waste”, “post-mining”), teaching (including popularization), land management (zones dedicated to extraction), philosophy (“needs” of ICT for Humans), history (current impacts of former supplies), law (how to legislate/regulate?)…

With his experience in various countries/projects, Johan Yans will expose some of these aspects, with a particular focus on the supply of geological resources for ICT needs. Olivier Vergeynst will then introduce the basics of Sustainable IT, highlighting some best practices to start reducing your impact as ICT professional or as simple user of digital devices and services.

Copper and gold prices were already high before the war in Ukraine. True for lots of minerals. Not for gas, that was influenced a lot by the war.

Each year we extract 92 billion tons of minerals each year. That is one swimming pool every two seconds. 60 percent of cobalt in the world is extracted from the DRC, Congo. A lot is done artisan ally, by hand.

The list of critical raw materials is rising, so materials that are important, in our view from Europe, but that have supply risks. Europe needs to import a lot.

International Resource Panel says we need to extract less minerals if we want to keep below 2 degrees warmup in 2060. But the expectation is that we will extract two times as much.

We can extract minerals in Belgium, there is a lot of zinc and lead. Of course we say: not in my back yard! There are a lot of downsides.

The ICT industry accounts for more green house gases than civil aviation. And between 60 and 90 percent of our equipment ends up in land fills, instead of being recycled.

We have an Institute for Sustainable IT. Impact is mostly for user equipment, that takes the most energy and minerals. Manufacturing an iPhone 11 taks 72 kg of carbon emissions, which is about 80 percent of the emissions for its total life cycle.

For data usage, the main impact comes from videos. Online video is 60 percent, 20% other videos, like meetings, 20 % other content. If you can choose a lower transmission rate on Netflix, that helps.

Sustainable IT is good for business. Get ready ahead of regulations, and you will have a chance to catch up with your competitors and go beyond them. Risk management: unsustainability is a vulnerability.

Choose for eco-design of digital services. First question: what can I avoid developing? Write efficient code. Reduce bandwidth. Divide infra needs. Extend the lifetime of your product: if your product can still run on older phones, you avoid the need for your users to buy a new phone. More clients. Business growth.

Demonstrate your engagement. You can become a member. Or just sign.

A few tips:

• Buy less equipment
• Reduce video streaming
• Limit use of emails, avoid sending it, avoid sending large attachments.
• Use https://www.printwhatyoulike.com/

Vincent Paulis: From IT to brewery

I grew up in Spa, studied IT in Liege. I like comic strips. My company is called Bridge IT. A few years ago I adapted my company to officially sell beer. Brasserie du Point, le Gout du Plaisir. If you would stop working in IT, what would you do? I followed a micro brewery training. And I have a comic from Renaud, a well known comic artist, on my beer.

Andreas Jung: Typesense integration for Plone

A new full text solution for Plone 6.

A long time ago I created Products.TextIndexNG3, but after a long time it died. I was searching for a decent replacement. I came across Typesense, a few fault-tolerant, full text search solution. Faceted search.

The browser gets javascript from Plone and then talks directly to the Typesense server, so it is very fast.

Every widget is configured through javascript, very customisable.

You can deploy it on-premise or in a private cloud, either as single binary or as Dicker,.

Code: https://github.com/zopyx/zopyx.typesense

Mikel Larreategi: volto-react-table-widget

We had a use case for a kind of datagridfield. Standard stuff worked okay, but hat if the user needs more than 1000 rows? The browser dies.

How about react-table? Really fast. We integrated it into Volto.

For bonus points: CSV export import.

See https://github.com/eea/volto-react-table-widget

Erico Andrei: collective.elasticsearch

Developed originally by Nathan van Gheem. The Brazilian electoral justice site uses this. We have been updating it to Python 3. Also integrates with Volto.Simple to configure. Specify which indexes you want to use for Elasticsearch, press a button, and it will migrate. You can even delete the indexes from the portal catalog.

See https://github.com/collective/collective.elasticsearch

Fred van Dijk: Improv wisdom

Useful self help for your personal life. And how to participate in the Plone Community.

Improv wisdom is a book from 2005 by Patricia Plan Madson. See http://improvwisdom.com/

Improvisation theater: earlier in life I thought: how can this possibly be fun?

I realised today that this is also an interesting take on how the Plone community works.

Ideas from the book:

1. Say yes. Be more adventurous. Don't say no by default.
2. Don't prepare. Give up planning.
3. Just show up. Just being somewhere enables things.
4. Start anywhere.
5. Be average. Dare to be dull. Be nothing special.
6. Read the book.

Philip Bauer: What's awesome?

A community-curated list of awesome Plone add-ons: https://github.com/collective/awesome-plone

A curated list of things that make Volto awesome: https://github.com/collective/awesome-volto

We need you to help with this list: create an issue or pull request to add packages to these lists, or to remove no longer relevant ones.

It is fine to add your own, you are good.

Jon Pentland: Volto use case

NSW Government released a design system In 2018, now third version, we updated it. Result is 29 new Volto components. Hundreds of variations.

Lessons learned:

• Time consuming to convert extern html/css into Volto components
• Lots of shadows of core were needed, which makes it hard to upgrade.
• Other than that, upgrading Volto is pretty painless.
• We have same idea as https://github.com/collective/collective.patchwatcher

See https://github.com/pretagov/nsw-design-system-plone6

Victor Fernandez De Alba: Auto login

There was some problem that I created a fix for as Chrome extension. Click it and you are logged in. Also works in Firefox.

See https://github.com/collective/plone6-autologin-extension

Jens Klein: pipx

When you install an executable Python based tool, for example cookie cutter, this pollutes your virtual environment. pipx creates virtual envs on the fly, one for each tool. It updates, unless you tell it not to. Just use e.g. pipx run black. Or pipx run --spec zest.releaser fullrelease if the command differs from the package name

Flip McFadden: POSKeyError: how not to fix this

This is a serious talk.

You could use backups, but those are for the weak. So you have a POSKeyError. I like to debug things. A lot of people tell you how to do things, for example this smart guy. Don't belief him, that would be boring.

So try stuff. Use RelStorage. Reindex. Get advice from community forums. Step six or so: invent a new word: Kadosectomy. We will remove a bucket from the BTree. Seventh step: manually bisect a BTree and delete the bucket. Remove the BTree. Replace it with a fresh copy. Your problem is now fixed, and you can let someone else fix the AttributeError.