Weblog

published Nov 03, 2021, last modified Nov 04, 2021

Chrissy Wainwright: State of the Plone Community

published Oct 25, 2021

Keynote talk by Chrissy Wainwright at the online Plone Conference 2021.

The community surrounding the Plone Foundation is an integral part in the success of Plone, and is what continually draws us to meet together for conferences and sprints.

Plone Foundation President Chrissy Wainwright talks about what the community has meant to her, the current state of the community, and what our hopes are for the future.

Each conference is a family reunion for us. It is also about swapping pants, borrowing flags, playing accordeon, drinking a barrel of saki, sword fights, or trying to get a katana on a plane.

Plone has a Steering Circle now, where team leaders and others meet, and discuss what is happening currently in Plone and where we are going. If you have questions for these meetings, please send them. The next one is next Friday during this conference.

https://training.plone.org has lots of training material, including the updated trainings from last weekend.

We have two podcasts now: the Plone podcast and the Plone Newsroom.

Sprints are meetings in person where we code, document, discuss. Covid has made this mostly impossible, but some online sprints were held. This online conference is being followed by people on a sprint in Sorrento, Italy.

My first talk was in 2009 about viewlets in Plone 3. I realized that due to my work for SixFeetUp I was learning parts of Plone that not everyone else knew. Since then I have done lots of talks, and have been on the Plone Foundation Board for seven years. My current term will be my last, so I can focus more on my family. I won't go anywhere: I will continue to use Plone.

Get involved! How can you help Plone?

Tips for gettings started:

  • Even small contributions are helpful.
  • Don't complain about things that do not work as you expect. Open an issue instead. Document a workaround.
  • Ignore the imposter syndrome. Lots of Plone people will be happy to help you.

I am looking forward to many more Plone memories.

Philip Bauer: Growing Pains: PosKeyErrors and Other Malaises

published Dec 11, 2020, last modified Jan 05, 2021

Talk at Plone Conference 2020

This talk is about the issues that you face when your project grows, the code base grows, the database grows, the problems grow. This is about the causes and some of the remedies.

Symptom 1: huge database

Cause 1: a huge number of revisions or versions.

Remedies:

  • Remove all versions and pack the database. When you migrate to a new Plone version, and you ask your client, they will usually be okay with this.
  • Manage or limit revisions. Easiest is to use collective.revisionmanager for this. Especially, revisions may have been left behind for content that no longer exists. You can easily remove it with this tool.
  • Disable versioning of Files. It is disabled by default, but maybe someone has switched it on.
  • Enable manual versioning instead of automatic. Then the editor needs to check a box when they make a major change that they want to be able to rollback.

Cause 2: no packing.

Remedy: just pack it. Use the zeopack script, which part of plone.recipe.zeoserver. Add a cronjob for this, weekly seems best for most sites.

Cause 3: unused content.

Remedy: delete it. You have to find it first. Of course no code can tell you which content is safe to delete. You could use statistics.py from collective.migrationhelpers to get an idea of where which content is.

Cause 4: the SearchableText index is huge

Remedies:

  • Use solr or elasticsearch and possibly remove the SearchableText index.
  • Don't index files. They are converted to text, but this may not be needed for your site.

Cause 5: large blobs For example, plone.de had a Linux iso image, which was huge.

Remedies:

  • Limit the upload size. You could do this in nginx/apache. Archetypes had something, you can likely do this in Dexterity too.
  • Get stats and remove or replace too large items.

Cause 6: aborted uploads (rare)

Remedy: check IAnnotations(portal).get('file_upload_map').

Symptom 2: slow site

Cause 1: unneeded full renders of content

Remedy: use Python in page templates. By default, page templates use path expressions like this: tal:define="foo context/foo". But this tries to render  foo as html if possible. Use foo python:context.foo instead.

Cause 2: wake up many objects

Remedies:

  • Always try to use brains and metadata. The difference is huge, also with Dexterity.
  • Listing 3000 brains: 0.2 seconds
  • Listing 3000 objects: 2 seconds
  • Same is true for Volto when you use the search-endpoint with fullobjects.

Of course most page templates in Plone will not list thousands of objects, but will be paginated. Still: just use brains, they are so much tastier.

Cause 3: no caching

Remedies:

  • Switch on the built-in caching
  • Add varnish
  • Manage the zeocache (that is a bit of science, ask the community)
  • Use memoize in your code.

Cause 4: hardware

Remedies:

  • Don't be cheap.
  • Buy enough ram to keep the database in memory.
  • Remember that your consulting time probably costs more than buying better hardware would.

Cause 5: slow code

Remedies:

  • Learn and use profiling. A very handy toy for that is py-spy. Sample use: sudo py-spy top --pid 12345
  • Do not call methods multiple times from templates. Call them once, store the result, and use this.

Cause 6: slow data sources

Remedies:

  • decouple, for example using redis or celery
  • Use your choice of async implementations
  • Use lazyloading of images if they come from outside of your Plone site.

Symptom 3: conflict errors

Conflict errors happen when two requests work at the same time and both change the same object. This is complicated, but Zope and the ZODB have built-in conflict resolution.

Cause 1: conflict resolving is not enabled. The zeoserver needs access to the same code that your zeoclient has, otherwise conflicts cannot be resolved and the transaction will be aborted.

Remedy: add all application code to the zeoserver:

[zeoserver]
eggs = ${buildout:eggs}

Cause 2: long running requests change data

Remedies:

  • Prevent writes.
  • If it takes long, do intermediate commits when possible.
  • Prevent crossfire: disable cronjobs and editors when a long request needs to run.
  • Use async. Talk to Asko about that probably.

Symptom 4: PosKeyErrors

Cause 1: missing blobs

Remedies:

  • Copy all blobs of course.
  • Use experimental.gracefulblobmissing in development to create dummy blobs where needed.
  • Find and delete afflicted content in a browser view.
  • There can be cases when you have two zeoclients and the syncing does not work well. Talk to Alessandro about that.

Symptom 5: broken data

Now for the really interesting part. These are errors like:

ModuleNotFoundError
AttributeError
ImportError
PostKeyError
BrokenObject

I could read you my whole blog post about zodb debugging.

Cause 1: code to unpickle som data is missing

Remedies:

  • Ignore the errors, if normal operation still works, and the site only has to stay up for a limited time, because zeopack probably also fails.
  • Fix it with a rename_dict. See zest.zodbupdate for some examples that are actually really useful. [Thanks! MvR]
  • Work around it with an alias_module patch, like plone.app.upgrade does in several cases. Then imports can work again.
  • Find out what and where broken objects are and then fix or remove them safely. Use zodbverify.

Steps for the last one:

  • Call bin/zodbverify -f var/filestorage/Data.fs to get all broken objects.
  • Pick one error type at a time, with an oid (object id) that has a problem.
  • Call bin/zodbverify -f var/filestorage/Data.fs -o -D to inspect one object and find out where it is referenced.
  • For the extra options, you should use the branch from my pull request, which I still have not finished yet, but it runs fine.
  • Remove or fix the object.
  • Important: make notes, write upgrade steps, keep the terminal log, because you will forget it and need it again.

To remove or fix the object, it helps to start the actual Plone site with some special zodbverify sauce:

./bin/instance zodbverify -f var/filestorage/Data.fs -o  -D

Then you can use your debugging skills to try and fix things. Note that after you fixed it, you need to commit the changes explicitly:

import transaction
transaction.commit()

Note that the bad object is still in the database, until you pack it.

Frequent culprits are IntIds and Relations, especially if you migrated from Archetypes to Dexterity. Using collective.relationhelpers you can clean this up:

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()

Symptom 6: bad code

Unreadable, untested, unused, undocumented, unmaintained, complicated, overly complex, too much code. If you can convince a client to not want a feature because they will only use it once, that is a win. Every line of code that is not written, is a good line of code.

Maik Derstappen: Plone 6 Theming with Diazo

published Dec 10, 2020

Talk at Plone Conference 2020

How does Plone theming look in classic UI?

  • html5 theme plus a mapping configuration
  • deploying themes as ZIP-file for shared hosting is possible
  • With Diazo you can map any Plone html to a static theme layout.

Separating frontend and backend theme. Don't reinvent the backend views! You could theme the backend, so for content editors and admins, but it looks fine, not needed. You should focus on the frontend layout for visitors. To use the default backend layout you can include backend.xml in your rules, with some conditions.

Diazo is not for everything! If the backend markup differs from what you need, do not try to solve it with Diazo or XSLT. Instead, fix the backend templates directly, most likely with a z3c.jbot override.

You can use theme fragments or browser views to add new templates. The theme fragments can also be used as tiles in Mosaic.

I like using SASS mixins. Say you have a

class="main-wrapper container">
. Cleaner would be
class="main-wrapper">
. You can do this with mixins with @include in your selector.

For more information on Diazo, see https://diazo.org.

New features in Plone 6

From the backend we get Bootstrap 5 compatible html. Result is that Bootstrap themes are easier to integrate in Plone 6.

You have custom CSS in the theming control panel, for small changes. This actually sneaked into Plone 5.2.2 as well.

We have simplified Diazo rules.

Create a theme with plonecli:

  • pip install plonecli
  • plonecli create addon plonetheme.yourtheme
  • cd plonetheme.yourtheme
  • plonecli add theme
  • plonecli build

Theme from this presentation will be published as collective.bunapisicuto when it is ready for you to inspect.

Stefan Antonelli: Plone 6 Theming from Scratch

published Dec 10, 2020

Talk at Plone Conference 2020

How to create a theme for Plone 6. Quite easy, because the templates use Bootstrap 5 classes. We build a theme from scratch, no Barceloneta, no Diazo.

First step is to create an empty plone_addon package with plonecli or mr.bob. For the questions you can answer: use Plone 5.2.1. We will switch later. My theme is plonetheme.munich.

I recommend to cleanup the standard package a bit. I remove tests, constraints for Plone 4 and 5. Check it out in the commits.

Now switch to extend Plone 6 and run the buildout.

You can add theme structure with a bob template, but I prefer creating my own.

Some interesting files:

  • package.json lists various tasks, especially the watch task.
  • In the theme manifest.cfg we more or less disable Diazo by emptying the rules line.
  • The compiled CSS and JavaScript are registered in registry.xml.

You can compile SASS to CSS using npm or yarn. Do yarn install in the top of your package. Later, with yarn dist you make it ready for production.

After these steps Plone is partially broken, or at least ugly. I do some basic fixes and it looks better.

For templates that you need to change you add z3c.jbot overrides. Personally I always kick out the "search in section" checkbox.

I don't like columns, but for this example I kept them. In most cases I need just one column. Plus maybe a side bar for portlets, but portlets must die.

With plonetheme.tokyo everything is Bootstrap, no columns, so no portlets, really fully responsive. This was the package where we built on Barceloneta Plone 5.2 and introduced lots of template overrides to put in Bootstrap. For Plone 6 we can just remove the overrides.

What about the toolbar? Yes, we dropped it. We bring editing features and navigation together. This is now a few feature: collective.sidebar. It is only one template to override. It works for Plone 5.2 at the moment, and I may work on it for Plone 6 during the sprints.

Question: is TTW still a viable path?

Answer: I like to concentrate on one path. I am not an expert in TTW theming. I switched to file system, except really small customizations. For small CSS customizations there is a field in the theming control panel.

Stefan Antonelli and Peter Holzer: Modernize Plone’s Classic UI

published Dec 10, 2020

Talk at Plone Conference 2020

What was new in Plone 5? We had beautiful new theme: Barceloneta. Diazo theming by default. We switched to CSS compilation with less.

During the Tokyo conference Stefan thought up Tokyo Theme. Clean responsive theme for Plone 5. Tons of overrides to tackle problems in Plone 5. Issue with navigation and editing on mobile we solved with collective.sidebar.

We had community discussions, especially during several Plone events. Everyone tried to use Bootstrap (components). First idea: map variables from Barceloneta to Bootstrap, because they have similar ideas using different terms.

We have PLIP to modernize markup in templates, and another PLIP to modernize the default theme: Barceloneta LTS. Forms using z3c.form are already using the new classes.

Make things easier: UI, development. Creating a modern UI for the web is complex. You need to support different devices, responsiveness. In Bootstrap there are patterns for most useful things.

Developer perspective: expect one way to do things. Developers should not have to worry about design. When busy in the backend, you should focus on Python, not on it looking nice and shiny. Don't think about markup, just use components. The good news: there is documentation. The Bootstrap documentation is our documentation.

What is new in Plone 6?

  • Volto is the default UI.
  • There still is the Classic UI with Barceloneta look and feel, but updated.
  • No TTW (through the web) theming.
  • But there is a textarea to add some simple CSS (already in 5.2.3).
  • Some CSS variables may be changeable TTW.
  • Finally jQuery 3

Bootstrap is still the most popular front-end framework. Well documented, tested and maintained. It is so easy to create stuff, I enjoy it a lot.

What is new in Bootstrap 5?

  • Improved overall look and feel.
  • Updated and extended the color system
  • Custom properties: css variables
  • SVG Icon library
  • Pure javascript
  • Dropped IE10 and IE11 support
  • Bootstrap 5 is currently alpha 3.
  • See https://v5.getbootstrap.com

Features: what do we get from these changes?

  • Core templates use Bootstrap 5 markup. Instead of overrides in plonetheme.tokyo, we have lots of branches for the actual packages.
  • All major templates have been touched already.
  • For the current state, see the unofficial demo at https://classic.plone.de
  • The Bootstrap documentation has lots of snippets than you can copy.
  • You don't need much more CSS on top of it if you paste most examples. We added a little for own components, like navigation.

Barceloneta appearance is fully customizable. It is basically on opinionated set of bootstrap variables. Every aspect can be changed with variables: colors, fonts, sizes, spacings, grid gutters, etc. There are overall properties, like shadowed, rounded, gradients. Just turn on or off.

Theming workflow. plonetheme.barceloneta will also be published as npm package. bobtemplates.plone will have a template for the new theming workflow. You can do quick and dirty customizations through the CSS overrides field in the theme controlpanel.

Diazo will still be there, will work as before. Some optimizations in the rules.xml to make content are customizations easier.

How to deal with icons? What if you want to change the content type icons? Used to be hard. Now we come up with the idea of an icon resolver. We decided to use the bootstrap icons. Icons are registered via GenericSetup, for example with a record name plone.icon.alarm pointing to an SVG. You can then override this in your own GS profile.

Example icon use:

You get get back an inline SVG or an image tag.

Note: all z3c form widgets in Plone are now in plone.app.z3cform, and not scattered over lots of packages.

We will restart our weekly Plone 6 Classic UI sprints, starting Januari 13 2021, 10:00 (UTC+1).