Weblog

published Nov 03, 2021 , last modified Nov 04, 2021

Stefan Antonelli and Peter Holzer: Theming Plone 6 Classic UI

published Oct 25, 2021 , last modified Nov 04, 2021

Talk by Stefan Antonelli and Peter Holzer at the online Plone Conference 2021.

We made a couple of talks for Plone Conference 2020. This talk is basically a summary of the story behind modernize Plone's classic UI + Theming based on this work. We're going to show how to create a theme for Plone 6 Classic UI based on Barceloneta and from scratch.

In discussions during several Plone events, we saw that everyone tried to use Bootstrap, including its components. We wanted to use bootstrap variables in Plone, instead of having our own definitions with slightly different names.

We and others made several PLIPs, Plone Improvement Proposals:

  • Modernize markup in templates
  • Modernize default theme, Barceloneta LTS
  • Modernize javascript to ES6

We want to make things easier. Creating a modern UI for the web is complex. You need to support different devices. Users expect things to work in a certain way, because they have seen it on many other sites.

From a developer perspective:

  • You expect one way to do things.
  • Developers should not have to worry about design, markup. You just want to use a component.
  • Find examples in (Bootstrap) documentation and just copy it.

In Plone 6, the default UI is Volto. But Classic UI will remain there for years, so it had to be updated. For starters: Bootstrap 5. It is the most used framework. If you don't use it, you miss out on a lot of fun.

  • There is support for custom properties, also known as CSS variables.
  • SVG icon library
  • vanilla javascript, no jQuery
  • dropped IE10 and IE11 support.

In Plone 6 Classic UI:

  • Templates updated to Bootstrap 5.
  • They use the Bootstrap components as documented.
  • Barceloneta theme uses an opinionated set of bootstrap variables.

Demo site is on https://6-classic.demo.plone.org/

TTW theming:

  • No more Through The Web compilation.
  • You can sill upload a zip with a theme.
  • Support for custom css via theming controlpanel, via a simple form which is already there in Plone 5.2.

Filesystem based theming:

  • plonetheme-barceloneta-base as npm package
  • We have updated bobtemplates.plone using that package. Use plonecli to create an add-on package and then let it add a theme to your package.
  • Three options: you inherit from Barceloneta or create from scratch or use Diazo.
  • Compile the scss with npm.
  • Diazo is still there and works like before.

Javascript:

  • The javascript no longer uses RequireJS.
  • We updated jQuery, although you do not need it much anymore.
  • No more TTW build.
  • Mockup was udpated to ES6, based on patternslib.
  • Resource Registries is there, but the TTW compilation is gone.
  • There are basically only bundles, no resources.
  • You no longer have to create manual timestamps. The webresource library creates automatic hashes instead.

How to deal with icons?

  • We use bootstrap icons by default.
  • Icons are loaded as inline SVG.
  • We have icons for content types, Plone UI icons, toolbar, mimetypes with fallback.
  • Registration is done in plone.staticresources, in XML.
  • Use in a template:

<tal:icon replace="structure icons.tag('love', tag_class='custom-class', tag_alt='foobar')" />

Can you insert icons in content?

  • You could copy SVG into HTML, though it is not so nice.
  • Planned is a TinyMCE pluging to select and add an icon.
  • But discussion is in progress.

Custom icons in Plone:

  • Try out collective.fontawesome
  • Register SVG icons as resource.
  • Add XML to register icon names.
  • Setup a profile for an icon font, but that is also part of discussion.

Further:

  • Typography and overall look is modernized.
  • Especially the Event, File and Image content type look nicer.
  • Control panel has much bigger icons.
  • We made sure it looks good on mobile devices.

How to theme:

Talk to us on Plone Discord in the #classic-ui room.

Chrissy Wainwright: State of the Plone Community

published Oct 25, 2021

Keynote talk by Chrissy Wainwright at the online Plone Conference 2021.

The community surrounding the Plone Foundation is an integral part in the success of Plone, and is what continually draws us to meet together for conferences and sprints.

Plone Foundation President Chrissy Wainwright talks about what the community has meant to her, the current state of the community, and what our hopes are for the future.

Each conference is a family reunion for us. It is also about swapping pants, borrowing flags, playing accordeon, drinking a barrel of saki, sword fights, or trying to get a katana on a plane.

Plone has a Steering Circle now, where team leaders and others meet, and discuss what is happening currently in Plone and where we are going. If you have questions for these meetings, please send them. The next one is next Friday during this conference.

https://training.plone.org has lots of training material, including the updated trainings from last weekend.

We have two podcasts now: the Plone podcast and the Plone Newsroom.

Sprints are meetings in person where we code, document, discuss. Covid has made this mostly impossible, but some online sprints were held. This online conference is being followed by people on a sprint in Sorrento, Italy.

My first talk was in 2009 about viewlets in Plone 3. I realized that due to my work for SixFeetUp I was learning parts of Plone that not everyone else knew. Since then I have done lots of talks, and have been on the Plone Foundation Board for seven years. My current term will be my last, so I can focus more on my family. I won't go anywhere: I will continue to use Plone.

Get involved! How can you help Plone?

Tips for gettings started:

  • Even small contributions are helpful.
  • Don't complain about things that do not work as you expect. Open an issue instead. Document a workaround.
  • Ignore the imposter syndrome. Lots of Plone people will be happy to help you.

I am looking forward to many more Plone memories.

Philip Bauer: Growing Pains: PosKeyErrors and Other Malaises

published Dec 11, 2020 , last modified Jan 05, 2021

Talk at Plone Conference 2020

This talk is about the issues that you face when your project grows, the code base grows, the database grows, the problems grow. This is about the causes and some of the remedies.

Symptom 1: huge database

Cause 1: a huge number of revisions or versions.

Remedies:

  • Remove all versions and pack the database. When you migrate to a new Plone version, and you ask your client, they will usually be okay with this.
  • Manage or limit revisions. Easiest is to use collective.revisionmanager for this. Especially, revisions may have been left behind for content that no longer exists. You can easily remove it with this tool.
  • Disable versioning of Files. It is disabled by default, but maybe someone has switched it on.
  • Enable manual versioning instead of automatic. Then the editor needs to check a box when they make a major change that they want to be able to rollback.

Cause 2: no packing.

Remedy: just pack it. Use the zeopack script, which part of plone.recipe.zeoserver. Add a cronjob for this, weekly seems best for most sites.

Cause 3: unused content.

Remedy: delete it. You have to find it first. Of course no code can tell you which content is safe to delete. You could use statistics.py from collective.migrationhelpers to get an idea of where which content is.

Cause 4: the SearchableText index is huge

Remedies:

  • Use solr or elasticsearch and possibly remove the SearchableText index.
  • Don't index files. They are converted to text, but this may not be needed for your site.

Cause 5: large blobs For example, plone.de had a Linux iso image, which was huge.

Remedies:

  • Limit the upload size. You could do this in nginx/apache. Archetypes had something, you can likely do this in Dexterity too.
  • Get stats and remove or replace too large items.

Cause 6: aborted uploads (rare)

Remedy: check IAnnotations(portal).get('file_upload_map').

Symptom 2: slow site

Cause 1: unneeded full renders of content

Remedy: use Python in page templates. By default, page templates use path expressions like this: tal:define="foo context/foo". But this tries to render  foo as html if possible. Use foo python:context.foo instead.

Cause 2: wake up many objects

Remedies:

  • Always try to use brains and metadata. The difference is huge, also with Dexterity.
  • Listing 3000 brains: 0.2 seconds
  • Listing 3000 objects: 2 seconds
  • Same is true for Volto when you use the search-endpoint with fullobjects.

Of course most page templates in Plone will not list thousands of objects, but will be paginated. Still: just use brains, they are so much tastier.

Cause 3: no caching

Remedies:

  • Switch on the built-in caching
  • Add varnish
  • Manage the zeocache (that is a bit of science, ask the community)
  • Use memoize in your code.

Cause 4: hardware

Remedies:

  • Don't be cheap.
  • Buy enough ram to keep the database in memory.
  • Remember that your consulting time probably costs more than buying better hardware would.

Cause 5: slow code

Remedies:

  • Learn and use profiling. A very handy toy for that is py-spy. Sample use: sudo py-spy top --pid 12345
  • Do not call methods multiple times from templates. Call them once, store the result, and use this.

Cause 6: slow data sources

Remedies:

  • decouple, for example using redis or celery
  • Use your choice of async implementations
  • Use lazyloading of images if they come from outside of your Plone site.

Symptom 3: conflict errors

Conflict errors happen when two requests work at the same time and both change the same object. This is complicated, but Zope and the ZODB have built-in conflict resolution.

Cause 1: conflict resolving is not enabled. The zeoserver needs access to the same code that your zeoclient has, otherwise conflicts cannot be resolved and the transaction will be aborted.

Remedy: add all application code to the zeoserver:

[zeoserver]
eggs = ${buildout:eggs}

Cause 2: long running requests change data

Remedies:

  • Prevent writes.
  • If it takes long, do intermediate commits when possible.
  • Prevent crossfire: disable cronjobs and editors when a long request needs to run.
  • Use async. Talk to Asko about that probably.

Symptom 4: PosKeyErrors

Cause 1: missing blobs

Remedies:

  • Copy all blobs of course.
  • Use experimental.gracefulblobmissing in development to create dummy blobs where needed.
  • Find and delete afflicted content in a browser view.
  • There can be cases when you have two zeoclients and the syncing does not work well. Talk to Alessandro about that.

Symptom 5: broken data

Now for the really interesting part. These are errors like:

ModuleNotFoundError
AttributeError
ImportError
PostKeyError
BrokenObject

I could read you my whole blog post about zodb debugging.

Cause 1: code to unpickle som data is missing

Remedies:

  • Ignore the errors, if normal operation still works, and the site only has to stay up for a limited time, because zeopack probably also fails.
  • Fix it with a rename_dict. See zest.zodbupdate for some examples that are actually really useful. [Thanks! MvR]
  • Work around it with an alias_module patch, like plone.app.upgrade does in several cases. Then imports can work again.
  • Find out what and where broken objects are and then fix or remove them safely. Use zodbverify.

Steps for the last one:

  • Call bin/zodbverify -f var/filestorage/Data.fs to get all broken objects.
  • Pick one error type at a time, with an oid (object id) that has a problem.
  • Call bin/zodbverify -f var/filestorage/Data.fs -o -D to inspect one object and find out where it is referenced.
  • For the extra options, you should use the branch from my pull request, which I still have not finished yet, but it runs fine.
  • Remove or fix the object.
  • Important: make notes, write upgrade steps, keep the terminal log, because you will forget it and need it again.

To remove or fix the object, it helps to start the actual Plone site with some special zodbverify sauce:

./bin/instance zodbverify -f var/filestorage/Data.fs -o  -D

Then you can use your debugging skills to try and fix things. Note that after you fixed it, you need to commit the changes explicitly:

import transaction
transaction.commit()

Note that the bad object is still in the database, until you pack it.

Frequent culprits are IntIds and Relations, especially if you migrated from Archetypes to Dexterity. Using collective.relationhelpers you can clean this up:

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()

Symptom 6: bad code

Unreadable, untested, unused, undocumented, unmaintained, complicated, overly complex, too much code. If you can convince a client to not want a feature because they will only use it once, that is a win. Every line of code that is not written, is a good line of code.

Maik Derstappen: Plone 6 Theming with Diazo

published Dec 10, 2020

Talk at Plone Conference 2020

How does Plone theming look in classic UI?

  • html5 theme plus a mapping configuration
  • deploying themes as ZIP-file for shared hosting is possible
  • With Diazo you can map any Plone html to a static theme layout.

Separating frontend and backend theme. Don't reinvent the backend views! You could theme the backend, so for content editors and admins, but it looks fine, not needed. You should focus on the frontend layout for visitors. To use the default backend layout you can include backend.xml in your rules, with some conditions.

Diazo is not for everything! If the backend markup differs from what you need, do not try to solve it with Diazo or XSLT. Instead, fix the backend templates directly, most likely with a z3c.jbot override.

You can use theme fragments or browser views to add new templates. The theme fragments can also be used as tiles in Mosaic.

I like using SASS mixins. Say you have a

class="main-wrapper container">
. Cleaner would be
class="main-wrapper">
. You can do this with mixins with @include in your selector.

For more information on Diazo, see https://diazo.org.

New features in Plone 6

From the backend we get Bootstrap 5 compatible html. Result is that Bootstrap themes are easier to integrate in Plone 6.

You have custom CSS in the theming control panel, for small changes. This actually sneaked into Plone 5.2.2 as well.

We have simplified Diazo rules.

Create a theme with plonecli:

  • pip install plonecli
  • plonecli create addon plonetheme.yourtheme
  • cd plonetheme.yourtheme
  • plonecli add theme
  • plonecli build

Theme from this presentation will be published as collective.bunapisicuto when it is ready for you to inspect.

Stefan Antonelli: Plone 6 Theming from Scratch

published Dec 10, 2020

Talk at Plone Conference 2020

How to create a theme for Plone 6. Quite easy, because the templates use Bootstrap 5 classes. We build a theme from scratch, no Barceloneta, no Diazo.

First step is to create an empty plone_addon package with plonecli or mr.bob. For the questions you can answer: use Plone 5.2.1. We will switch later. My theme is plonetheme.munich.

I recommend to cleanup the standard package a bit. I remove tests, constraints for Plone 4 and 5. Check it out in the commits.

Now switch to extend Plone 6 and run the buildout.

You can add theme structure with a bob template, but I prefer creating my own.

Some interesting files:

  • package.json lists various tasks, especially the watch task.
  • In the theme manifest.cfg we more or less disable Diazo by emptying the rules line.
  • The compiled CSS and JavaScript are registered in registry.xml.

You can compile SASS to CSS using npm or yarn. Do yarn install in the top of your package. Later, with yarn dist you make it ready for production.

After these steps Plone is partially broken, or at least ugly. I do some basic fixes and it looks better.

For templates that you need to change you add z3c.jbot overrides. Personally I always kick out the "search in section" checkbox.

I don't like columns, but for this example I kept them. In most cases I need just one column. Plus maybe a side bar for portlets, but portlets must die.

With plonetheme.tokyo everything is Bootstrap, no columns, so no portlets, really fully responsive. This was the package where we built on Barceloneta Plone 5.2 and introduced lots of template overrides to put in Bootstrap. For Plone 6 we can just remove the overrides.

What about the toolbar? Yes, we dropped it. We bring editing features and navigation together. This is now a few feature: collective.sidebar. It is only one template to override. It works for Plone 5.2 at the moment, and I may work on it for Plone 6 during the sprints.

Question: is TTW still a viable path?

Answer: I like to concentrate on one path. I am not an expert in TTW theming. I switched to file system, except really small customizations. For small CSS customizations there is a field in the theming control panel.