Plan:

• Brief introduction to each framework
• Followed by discussion focused on use cases
• Maurits will live blog (thanks!)
• Framework vs. Product - let’s not worry about that!

For each system:

• What use cases is it well suited to
• What use cases is it poorly suited to

Discussion:

• Focus on client use cases and what frameworks would be a good or poor fit
• You can also ask questions!

Live blogging.

Use cases:

• Non-profit with custom forms and import/export
• Concert venue with strict event listing and advanced settings.
• NGO needs a database application for their highly confidential data.
• Generate a map out of data, and store extra information, like images for the data.
• Three-person company with basically static pages, non-technical users.
• Real-time application with websockets.
• A large company wants to monitor how well their customer service staff are doing. Strict form with statistics on it.

Pyramid:

• Best for APIs, not really CMS or websites, but applications
• Start small, grow
• Complex non-profit use case: you would have to build everything yourself.
• Event listing: we have a site that manages stuff for music bands as example, you just need some endpoints. Maybe websauna on top.
• NGO privacy: security really good, and simple enough to understand the full system. You can fully build your own fine grained security system. UI is a lot of work.
• 100 person company wants intranet with various content, installable by local IT team.
• Intranet: try to create a good solution for multiple clients, not tailored to one in a consulting project. Pyramid is good for building such products.
• Map: pyramid does not care which database you use. In a project we are mapping electical grids. In Guillotina they think they are nice with AsyncIO, but it makes their whole codebase harder to understand.
• Real-time: we had websockets for a year, but it was too slow, so we rewrote it in Go-Lang.
• Form statistics: you can create a form, store it in a database, export it, fine.

Django:

• Aimed at 80% of use cases, simple auth system
• Good for CRUD apps, smooth beginning
• Non-profit: export/import is good, we can do forms. Sharing access gets tricky
• Event listing: Django can do it, but just use Wordpress. But the technical user can do this in the admin interface.
• NGO privacy: permissions are too simple OOTB, not per object. You may want end-to-end encryption, which no framework offers. Limited built-in audit log, nice start.
• Intranet: not good fit for Plone for the people who do not often work with it. Django has easier CMSes for this. The expenses claim form could be easier to integrate into Django. Deployment just needs Postgres. DjangoCMS is Plone Light. Wagtail is WordPress Plus.
• Map: any of the frameworks except Plone can do the geo stuff in postgres. Images just on the file system. Static and uploaded files are handled fine by Django, can also be in the cloud.
• Real-time: not many Django users will care about websockets, will never be the focus.
• Form statistics: you can but I would not do it. This feels like a custom web application. Do it in any framework.

Guillotina:

• Small framework, scale from small to big
• Non-profit with forms: decent fit, use JSON schema
• Event listing: go to wix.com or something, it seems too simple
• NGO privacy: Good fit, permissions are no problem for this. You need to build the full UI. Okay, you may be able to start with Volto as UI, but that needs integration in the backend.
• Intranet: backend can handle it, need to build UI, deployment can be with docker and kubernetes.
• Map: for the large files, Guillotina wins because of AsyncIO.
• Real-time: we use websockets just fine.
• Form statistics: you can build an app, depending on your UI skills.

Plone:

• Good when you have different use groups with complex requirements for security.
• Edit interface is doable, but less so on mobile.
• Import/export complicated, but you can hire Jens.
• Difficult to find skilled people.
• Non-profit with forms: really good fit, end-users can create the forms. The import/export would need help from a provider.
• Event listing: do not bother with Plone. It sounds more relational, Plone would be overkill.
• NGO privacy: you need deep understanding of Plone security, hierarchy for storing data. Audit logging with add-on.
• Intranet: Plone is fine. You may need to help with installing at first, but it will keep running once installed.
• Small almost static site: none of us.
• Real-time: once Asko is done with his ZServer improvements, we can do it.
• Form statistics: do not use Plone. Use sed, awk, perl. Technically you could use some add-ons.

What is PAS? It is the Pluggable Authentication Service from Zope and Plone. It manages everything related to users and groups, like authentication, permissions, searching. It can get or set information from your site or the request or an external service like LDAP.

It uses plugins, so you can register your own. You can see a lot of those in the ZMI at the Zope root, at http://localhost:8080/acl_users/plugins/manage_active.

There are about twenty different plugin types, and they interact with each other.

In Plone we have Products.PlonePAS. It adds Plone-specific plugins and types in PAS, for example the local roles plugins. Local roles are regular parts of Zope, but not pluggable there. So this is at http://localhost:8080/Plone/acl_users/plugins/manage_active

So this is the basis to manage security, users and groups, using plugins that can be activated, deactivated, and ordered, and these plugins interact with each other.

Now let's go to the kitchen and make a plugin. We want our Plone Intranet to use an LDAP plugin. At the time, we started using Products.PloneLDAP, which builds on a few other Zope packages. (Currently pas.plugins.ldap is the more modern choice.)

We also wanted Products.membrane: this allows to create content that works as user or group. It uses a separate membrane_catalog, a sort-of copy of the portal_catalog, for better or worse.

We wanted collective.workspace to manage roles through groups instead of sharing.

And Plone Intranet itself had some extra sauce (plugins) on top. And important: caching, for performance.

It worked fine! But after a while it slowed down. We had many calls to LDAP, in the wrong way, even with the cache in place. We were using the membrane catalog too much, causing the catalog queue to flush, causing a slowdown in the standard portal_catalog search. We were also writing on SOLR, again external, which did not help for performance.

So what was the reason of the bottleneck? The big deal is when PAS does not find what it is looking for. It then tries all relevant plugins, which are intertwined. For example it finds a workspace group, and needlessly goes to LDAP to search for this group there, where it will not be found, etcetera.

So maybe we could store the LDAP information in Plone. We decided to sync LDAP in the Plone objects.

And then we also patched PAS. Finding a group results in trying to get the properties, parent groups, roles. Our patched PAS skipped plugins that looked unrelated.

There was another problem: a custom local role manager. There were users that belonged to lots of workspaces. So also to lots of groups. So role assignment was slow: you need to get all roles of all groups. We replaced Plone's local role manager with a custom one.

Another plugin is the recursive groups plugin. Standard in Plone. This can be really expensive. For each of those thousands of groups you would check if they have a parent group. We replaced this with a utility that could handle this fast, using graphs with networkx. With this, we could replace all our group plugins, especially the recursive groups plugin.

Possible problem: in the graph we had 20.000 nodes, 25.000 edges, 25 relations. But creating the graph took 50 milliseconds, so that was okay. The nodes are strings like principal_type:UID, for example collective.workspaces:Admins. We stored the information in OOBTrees: uid2path, path2name, name2uid, and an OOTreeSet called edges. And then nx.DiGraph(uid2path) to create the graph. This helped speed things up a lot.

To keep the data structure up to date, we use events.

To recap:

• remove external connections when possible
• know the limitations of your plugins
• you can patch PAS to avoid some plugins
• you can use a custom group plugin

Further ideas:

• lazily create collective.workspace groups
• lazily fetch group and user properties and roles

Takeaways:

• PAS is great, but mixing and abusing plugins can be deadly
• Sometimes replace plugins with a new one that does exactly what you need.

Three PAS plugins:

• collective.denyroles
• pas.plugins.headers
• collective.contentgroups

At KitConcept we have some websites that need to be deployed in Plone 5.2. We could have used collective.transmogrifier to migrate from 4.3 to 5+. And from 5.1 to 5.2 migrate the ZODB from Python 2.7 to Python 3. Transmogrifier is also an option for that part.

Why would we use transmogrifier? There are many generic pipelines available for common cases. Flexibility to deal with different use cases. And it is actually a brilliant wayto use the iterator design pattern. Every piece of the pipeline changes a small thing, and hands the information over to the next piece.

Clients:

• Large university site
• High-profile government client
• Large research institution client website

Challenge: go to Python 3, Plone 5.2 and Volto. With Volto, they spare a migration from Plone 5 to 6, at least partially. Volto is a way to sell the client a Python 3 upgrade: it is the right time to use Volto.

We use Jenkins to test the migration.

General backend things:

• Old ATTopics to core Collections. Use collective.jsonify to export it, in a way that already looks more like a Collection.
• Rich text: migrate to Volto blocks.
• Post migration: collective.cover needed special handling.

What we polish to enter in Volto land:

• We use collective folderish types.
• deal with default pages
• Convert RichText to Volto DraftJS
• easily point to old website when content is not imported
• fix some urls (we want to use resolveuid)
• simple folders we should turn info page with listing block
• simple collections we should turn info page with collection block

Question: can you open source the module that handles the RichText to Volto blocks migration?

Answer: yes, it is actually just 38 lines with React in Node. I tried to do it in Python, but could not get it to work. And we can sprint on it.