Weblog

published Nov 03, 2021, last modified Nov 04, 2021

Maik Derstappen: Efficient Backend development for Plone

published Oct 25, 2021

Talk by Maik Derstappen at the online Plone Conference 2021.

How can we efficiently develop a Plone backend? What do we expect from the tooling?

  • Should be easy to adopt for new developers
  • reduced development time
  • following best practices
  • provide test infrastructure and basic tests

These are our two friends:

  • plonecli
  • plone-snippets for VS Code

plonecli is a command line tool, installed with pip together with bobtemplates.plone, here version 6.0b10. With it, you iteratively enhance your add-on.

Let's create a restaurant add-on with a content type for the menu data, a vocabulary for the menu category, control panel, plus a bit more.

Demo:

plonecli create addon collective.restaurant
cd collective.restaurant/
plonecli add controlpanel

Open Visual Studio Code (my preferred editor) and edit the created controlpanel.py. Add a field: a list of categories. Now add a vocabulary for this:

plonecli add vocabulary

Edit menu_category.py to let this get the contents of the controlpanel field from the registry.

Add two content types: Menu and Menu Item:

plonecli add content_type

Edit menu_item.py. When you have plone-vs-snippets installed in VS Code, it is easy to add fields. Add a photo (NamedBlobImage), menu_category (Choice with your menu vocabulary name).

Edit the types/*.xml files to enable behaviors like plone.basic and plone.namefromtitle.

Now get a local environment, mostly by running buildout, and start the instance, with:

plonecli build
plonecli serve

On http://localhost:8080 create a Plone Site and activate collective.restaurant and plone.restapi. Go to the Restaurant control panel and add a few categories, like Starters, Main, Desserts. Now you can add a Menu folder and add menu items in various categories.

The basic data is available through the REST api. But we want more. Add a service RelatedDishes:

plonecli add restapi service

Edit api/services/related_dishes/get.py and use api.content.find to look for other Menu Items with the same category.

To make searching easier, edit profiles/default/catalog.xml. With the Plone VS code snippets, you can agains easily add a catalog index here. We will need an upgrade step to apply this:

plonecli add upgrade_step

Restart, reindex, and it works.

Please help by contributing templates to bobtemplates.plone or create an issue when you think a template can be improved. Or create an own similar package, like bobtemplates.migration.

Ideas for the future:

  • Rest API support question in addon template
  • Rest API sub-templates for serializer and deserializer
  • support for Mosaic tiles
  • graphical UI (ncurses) to make choices easier

Kim Paulissen: Revamping the plone training and documentation setup for plone 6

published Oct 25, 2021

Talk by Kim Paulissen at the online Plone Conference 2021.

I have been working alongside of the documentation training teams the past months. In this talk I will tell you about the past, present and future of these. Of course, I follow this with a call to contribute!

I am just a messenger. Discussions can get heated between developers when talking about how to document things. Don't shoot the messenger.

https://plone.org should use Plone 6, with default frontend Volto. Rikupekka Oksanen will talk about that tomorrow.

Plone documentation

In the past, the documentation has been written in ReStructuredText, which is a markup language mostly used in the Python community. It was turned into readable web pages using Sphinx, a Python tool.

The main current contents, with target groups:

  • Working with content: for users
  • Adapting and extending Plone: for power users
  • Installing, managing and updating Plone: for sysadmins
  • Developing for Plone: for developers

Now to the present, or what is presently being created. Technology:

  • Markdown as markup language / syntax
  • Docusaurus as React based static site generator

Why this shift?

  • Plone 6 is a headless CMS with Volto (React) as default frontend.
  • Markdown is easier for new (and old) contributors.
  • Easy preview on GitHub and in an IDE.
  • For the people who need to create the new documentation focusing on Volto, they do not need to switch tech stacks.
  • In this way you can focus on the content.

We are not there yet. Various people are working on various branches:

The documentation is still setup to to separated into the two frontend options: Volto and Classic.

All content will be made from scratch. There was an enormous set of docs, some very old, and completely different from what we need now. The old docs will still be kept available.

Daydreaming for the future:

  • up to date documentation that is intuitive for all different parts of our setup
  • improved options for contributing
  • good CI/CD, resulting in changes that get to the live site fast

Plone training

For https://training.plone.org in the past we were using the same technology as docs.plone.org: ReStructuredText and Sphinx.

In the present, we moved to Markdown. Well, to MyST actually. This has most benefits of ReStructuredText added inside Markdown. If you don't know RST, you can simply write Markdown.

It still uses Sphinx, but with Markdown. We use the sphinx-book-theme.

The training team has restructured all the trainings. All were migrated to MyST.

Daydreaming for the future:

  • use the same tech stack for both docs and training
  • integrated following a good documentation system

Stefan Antonelli and Peter Holzer: Theming Plone 6 Classic UI

published Oct 25, 2021, last modified Nov 04, 2021

Talk by Stefan Antonelli and Peter Holzer at the online Plone Conference 2021.

We made a couple of talks for Plone Conference 2020. This talk is basically a summary of the story behind modernize Plone's classic UI + Theming based on this work. We're going to show how to create a theme for Plone 6 Classic UI based on Barceloneta and from scratch.

In discussions during several Plone events, we saw that everyone tried to use Bootstrap, including its components. We wanted to use bootstrap variables in Plone, instead of having our own definitions with slightly different names.

We and others made several PLIPs, Plone Improvement Proposals:

  • Modernize markup in templates
  • Modernize default theme, Barceloneta LTS
  • Modernize javascript to ES6

We want to make things easier. Creating a modern UI for the web is complex. You need to support different devices. Users expect things to work in a certain way, because they have seen it on many other sites.

From a developer perspective:

  • You expect one way to do things.
  • Developers should not have to worry about design, markup. You just want to use a component.
  • Find examples in (Bootstrap) documentation and just copy it.

In Plone 6, the default UI is Volto. But Classic UI will remain there for years, so it had to be updated. For starters: Bootstrap 5. It is the most used framework. If you don't use it, you miss out on a lot of fun.

  • There is support for custom properties, also known as CSS variables.
  • SVG icon library
  • vanilla javascript, no jQuery
  • dropped IE10 and IE11 support.

In Plone 6 Classic UI:

  • Templates updated to Bootstrap 5.
  • They use the Bootstrap components as documented.
  • Barceloneta theme uses an opinionated set of bootstrap variables.

Demo site is on https://6-classic.demo.plone.org/

TTW theming:

  • No more Through The Web compilation.
  • You can sill upload a zip with a theme.
  • Support for custom css via theming controlpanel, via a simple form which is already there in Plone 5.2.

Filesystem based theming:

  • plonetheme-barceloneta-base as npm package
  • We have updated bobtemplates.plone using that package. Use plonecli to create an add-on package and then let it add a theme to your package.
  • Three options: you inherit from Barceloneta or create from scratch or use Diazo.
  • Compile the scss with npm.
  • Diazo is still there and works like before.

Javascript:

  • The javascript no longer uses RequireJS.
  • We updated jQuery, although you do not need it much anymore.
  • No more TTW build.
  • Mockup was udpated to ES6, based on patternslib.
  • Resource Registries is there, but the TTW compilation is gone.
  • There are basically only bundles, no resources.
  • You no longer have to create manual timestamps. The webresource library creates automatic hashes instead.

How to deal with icons?

  • We use bootstrap icons by default.
  • Icons are loaded as inline SVG.
  • We have icons for content types, Plone UI icons, toolbar, mimetypes with fallback.
  • Registration is done in plone.staticresources, in XML.
  • Use in a template:

<tal:icon replace="structure icons.tag('love', tag_class='custom-class', tag_alt='foobar')" />

Can you insert icons in content?

  • You could copy SVG into HTML, though it is not so nice.
  • Planned is a TinyMCE pluging to select and add an icon.
  • But discussion is in progress.

Custom icons in Plone:

  • Try out collective.fontawesome
  • Register SVG icons as resource.
  • Add XML to register icon names.
  • Setup a profile for an icon font, but that is also part of discussion.

Further:

  • Typography and overall look is modernized.
  • Especially the Event, File and Image content type look nicer.
  • Control panel has much bigger icons.
  • We made sure it looks good on mobile devices.

How to theme:

Talk to us on Plone Discord in the #classic-ui room.

Chrissy Wainwright: State of the Plone Community

published Oct 25, 2021

Keynote talk by Chrissy Wainwright at the online Plone Conference 2021.

The community surrounding the Plone Foundation is an integral part in the success of Plone, and is what continually draws us to meet together for conferences and sprints.

Plone Foundation President Chrissy Wainwright talks about what the community has meant to her, the current state of the community, and what our hopes are for the future.

Each conference is a family reunion for us. It is also about swapping pants, borrowing flags, playing accordeon, drinking a barrel of saki, sword fights, or trying to get a katana on a plane.

Plone has a Steering Circle now, where team leaders and others meet, and discuss what is happening currently in Plone and where we are going. If you have questions for these meetings, please send them. The next one is next Friday during this conference.

https://training.plone.org has lots of training material, including the updated trainings from last weekend.

We have two podcasts now: the Plone podcast and the Plone Newsroom.

Sprints are meetings in person where we code, document, discuss. Covid has made this mostly impossible, but some online sprints were held. This online conference is being followed by people on a sprint in Sorrento, Italy.

My first talk was in 2009 about viewlets in Plone 3. I realized that due to my work for SixFeetUp I was learning parts of Plone that not everyone else knew. Since then I have done lots of talks, and have been on the Plone Foundation Board for seven years. My current term will be my last, so I can focus more on my family. I won't go anywhere: I will continue to use Plone.

Get involved! How can you help Plone?

Tips for gettings started:

  • Even small contributions are helpful.
  • Don't complain about things that do not work as you expect. Open an issue instead. Document a workaround.
  • Ignore the imposter syndrome. Lots of Plone people will be happy to help you.

I am looking forward to many more Plone memories.

Philip Bauer: Growing Pains: PosKeyErrors and Other Malaises

published Dec 11, 2020, last modified Jan 05, 2021

Talk at Plone Conference 2020

This talk is about the issues that you face when your project grows, the code base grows, the database grows, the problems grow. This is about the causes and some of the remedies.

Symptom 1: huge database

Cause 1: a huge number of revisions or versions.

Remedies:

  • Remove all versions and pack the database. When you migrate to a new Plone version, and you ask your client, they will usually be okay with this.
  • Manage or limit revisions. Easiest is to use collective.revisionmanager for this. Especially, revisions may have been left behind for content that no longer exists. You can easily remove it with this tool.
  • Disable versioning of Files. It is disabled by default, but maybe someone has switched it on.
  • Enable manual versioning instead of automatic. Then the editor needs to check a box when they make a major change that they want to be able to rollback.

Cause 2: no packing.

Remedy: just pack it. Use the zeopack script, which part of plone.recipe.zeoserver. Add a cronjob for this, weekly seems best for most sites.

Cause 3: unused content.

Remedy: delete it. You have to find it first. Of course no code can tell you which content is safe to delete. You could use statistics.py from collective.migrationhelpers to get an idea of where which content is.

Cause 4: the SearchableText index is huge

Remedies:

  • Use solr or elasticsearch and possibly remove the SearchableText index.
  • Don't index files. They are converted to text, but this may not be needed for your site.

Cause 5: large blobs For example, plone.de had a Linux iso image, which was huge.

Remedies:

  • Limit the upload size. You could do this in nginx/apache. Archetypes had something, you can likely do this in Dexterity too.
  • Get stats and remove or replace too large items.

Cause 6: aborted uploads (rare)

Remedy: check IAnnotations(portal).get('file_upload_map').

Symptom 2: slow site

Cause 1: unneeded full renders of content

Remedy: use Python in page templates. By default, page templates use path expressions like this: tal:define="foo context/foo". But this tries to render  foo as html if possible. Use foo python:context.foo instead.

Cause 2: wake up many objects

Remedies:

  • Always try to use brains and metadata. The difference is huge, also with Dexterity.
  • Listing 3000 brains: 0.2 seconds
  • Listing 3000 objects: 2 seconds
  • Same is true for Volto when you use the search-endpoint with fullobjects.

Of course most page templates in Plone will not list thousands of objects, but will be paginated. Still: just use brains, they are so much tastier.

Cause 3: no caching

Remedies:

  • Switch on the built-in caching
  • Add varnish
  • Manage the zeocache (that is a bit of science, ask the community)
  • Use memoize in your code.

Cause 4: hardware

Remedies:

  • Don't be cheap.
  • Buy enough ram to keep the database in memory.
  • Remember that your consulting time probably costs more than buying better hardware would.

Cause 5: slow code

Remedies:

  • Learn and use profiling. A very handy toy for that is py-spy. Sample use: sudo py-spy top --pid 12345
  • Do not call methods multiple times from templates. Call them once, store the result, and use this.

Cause 6: slow data sources

Remedies:

  • decouple, for example using redis or celery
  • Use your choice of async implementations
  • Use lazyloading of images if they come from outside of your Plone site.

Symptom 3: conflict errors

Conflict errors happen when two requests work at the same time and both change the same object. This is complicated, but Zope and the ZODB have built-in conflict resolution.

Cause 1: conflict resolving is not enabled. The zeoserver needs access to the same code that your zeoclient has, otherwise conflicts cannot be resolved and the transaction will be aborted.

Remedy: add all application code to the zeoserver:

[zeoserver]
eggs = ${buildout:eggs}

Cause 2: long running requests change data

Remedies:

  • Prevent writes.
  • If it takes long, do intermediate commits when possible.
  • Prevent crossfire: disable cronjobs and editors when a long request needs to run.
  • Use async. Talk to Asko about that probably.

Symptom 4: PosKeyErrors

Cause 1: missing blobs

Remedies:

  • Copy all blobs of course.
  • Use experimental.gracefulblobmissing in development to create dummy blobs where needed.
  • Find and delete afflicted content in a browser view.
  • There can be cases when you have two zeoclients and the syncing does not work well. Talk to Alessandro about that.

Symptom 5: broken data

Now for the really interesting part. These are errors like:

ModuleNotFoundError
AttributeError
ImportError
PostKeyError
BrokenObject

I could read you my whole blog post about zodb debugging.

Cause 1: code to unpickle som data is missing

Remedies:

  • Ignore the errors, if normal operation still works, and the site only has to stay up for a limited time, because zeopack probably also fails.
  • Fix it with a rename_dict. See zest.zodbupdate for some examples that are actually really useful. [Thanks! MvR]
  • Work around it with an alias_module patch, like plone.app.upgrade does in several cases. Then imports can work again.
  • Find out what and where broken objects are and then fix or remove them safely. Use zodbverify.

Steps for the last one:

  • Call bin/zodbverify -f var/filestorage/Data.fs to get all broken objects.
  • Pick one error type at a time, with an oid (object id) that has a problem.
  • Call bin/zodbverify -f var/filestorage/Data.fs -o -D to inspect one object and find out where it is referenced.
  • For the extra options, you should use the branch from my pull request, which I still have not finished yet, but it runs fine.
  • Remove or fix the object.
  • Important: make notes, write upgrade steps, keep the terminal log, because you will forget it and need it again.

To remove or fix the object, it helps to start the actual Plone site with some special zodbverify sauce:

./bin/instance zodbverify -f var/filestorage/Data.fs -o  -D

Then you can use your debugging skills to try and fix things. Note that after you fixed it, you need to commit the changes explicitly:

import transaction
transaction.commit()

Note that the bad object is still in the database, until you pack it.

Frequent culprits are IntIds and Relations, especially if you migrated from Archetypes to Dexterity. Using collective.relationhelpers you can clean this up:

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()

Symptom 6: bad code

Unreadable, untested, unused, undocumented, unmaintained, complicated, overly complex, too much code. If you can convince a client to not want a feature because they will only use it once, that is a win. Every line of code that is not written, is a good line of code.