Weblog

published Nov 03, 2021 , last modified Nov 04, 2021

Alessandro Pisa: PAS adventures

published Oct 25, 2019

Talk by Alessandro Pisa at the Plone Conference 2019 in Ferrara.

What is PAS? It is the Pluggable Authentication Service from Zope and Plone. It manages everything related to users and groups, like authentication, permissions, searching. It can get or set information from your site or the request or an external service like LDAP.

It uses plugins, so you can register your own. You can see a lot of those in the ZMI at the Zope root, at http://localhost:8080/acl_users/plugins/manage_active.

There are about twenty different plugin types, and they interact with each other.

In Plone we have Products.PlonePAS. It adds Plone-specific plugins and types in PAS, for example the local roles plugins. Local roles are regular parts of Zope, but not pluggable there. So this is at http://localhost:8080/Plone/acl_users/plugins/manage_active

So this is the basis to manage security, users and groups, using plugins that can be activated, deactivated, and ordered, and these plugins interact with each other.

Now let's go to the kitchen and make a plugin. We want our Plone Intranet to use an LDAP plugin. At the time, we started using Products.PloneLDAP, which builds on a few other Zope packages. (Currently pas.plugins.ldap is the more modern choice.)

We also wanted Products.membrane: this allows to create content that works as user or group. It uses a separate membrane_catalog, a sort-of copy of the portal_catalog, for better or worse.

We wanted collective.workspace to manage roles through groups instead of sharing.

And Plone Intranet itself had some extra sauce (plugins) on top. And important: caching, for performance.

It worked fine! But after a while it slowed down. We had many calls to LDAP, in the wrong way, even with the cache in place. We were using the membrane catalog too much, causing the catalog queue to flush, causing a slowdown in the standard portal_catalog search. We were also writing on SOLR, again external, which did not help for performance.

So what was the reason of the bottleneck? The big deal is when PAS does not find what it is looking for. It then tries all relevant plugins, which are intertwined. For example it finds a workspace group, and needlessly goes to LDAP to search for this group there, where it will not be found, etcetera.

So maybe we could store the LDAP information in Plone. We decided to sync LDAP in the Plone objects.

And then we also patched PAS. Finding a group results in trying to get the properties, parent groups, roles. Our patched PAS skipped plugins that looked unrelated.

There was another problem: a custom local role manager. There were users that belonged to lots of workspaces. So also to lots of groups. So role assignment was slow: you need to get all roles of all groups. We replaced Plone's local role manager with a custom one.

Another plugin is the recursive groups plugin. Standard in Plone. This can be really expensive. For each of those thousands of groups you would check if they have a parent group. We replaced this with a utility that could handle this fast, using graphs with networkx. With this, we could replace all our group plugins, especially the recursive groups plugin.

Possible problem: in the graph we had 20.000 nodes, 25.000 edges, 25 relations. But creating the graph took 50 milliseconds, so that was okay. The nodes are strings like principal_type:UID, for example collective.workspaces:Admins. We stored the information in OOBTrees: uid2path, path2name, name2uid, and an OOTreeSet called edges. And then nx.DiGraph(uid2path) to create the graph. This helped speed things up a lot.

To keep the data structure up to date, we use events.

To recap:

  • remove external connections when possible
  • know the limitations of your plugins
  • you can patch PAS to avoid some plugins
  • you can use a custom group plugin

Further ideas:

  • lazily create collective.workspace groups
  • lazily fetch group and user properties and roles

Takeaways:

  • PAS is great, but mixing and abusing plugins can be deadly
  • Sometimes replace plugins with a new one that does exactly what you need.

Maurits van Rees: 3 authentication add-ons

published Oct 25, 2019

Lightning talk by Maurits van Rees at the Plone Conference 2019 in Ferrara.

Three PAS plugins:

  • collective.denyroles
  • pas.plugins.headers
  • collective.contentgroups

collective.denyroles

Deny access to roles like Manager and Editor

  • Use case: Manager only logs in to edit-domain, not live site.
  • By default deny access to Manager, Editor, etc.
  • env DENY_ROLES=0 to disable
  • or Apache/nginx header X_DONT_CHECK_ROLES
  • Actually not a plugin, but a patch.

Code: https://github.com/collective/collective.denyroles/

pas.plugins.headers

PAS plugin for authentication based on request headers.

  • Use case: Apache/nginx adds SAML headers to requests.

  • Configuration in ZMI or profiles/default/pas.plugins.headers.json:

    {
        "userid_header": "uid",
        "required_headers": ["uid"],
        "roles_header": "roles",
        "allowed_roles": ["student", "teacher"],
        "deny_unauthorized": true,
        "redirect_url": "https://maurits.vanrees.org",
        "memberdata_to_header": [
            "fullname|HEADER_firstname HEADER_lastname"
        ]
    }
    

Code: https://github.com/collective/pas.plugins.headers/

collective.contentgroups

Plone PAS plugin for content as groups.

  • Use case: create content item that works as a group.
  • dexterity behavior
  • No Products.membrane, no Products.remember, no dexterity.membrane.
  • No separate membrane_catalog.
  • Only groups, not users.
  • No multiple inheritance, just AccessControl.users.BasicUser.

Code: https://github.com/collective/collective.contentgroups

Rodrigo Ferreira de Souza: Data migration to Plone 5.2 and Volto

published Oct 25, 2019

Talk by Rodrigo Ferreira de Souza at the Plone Conference 2019 in Ferrara.

At KitConcept we have some websites that need to be deployed in Plone 5.2. We could have used collective.transmogrifier to migrate from 4.3 to 5+. And from 5.1 to 5.2 migrate the ZODB from Python 2.7 to Python 3. Transmogrifier is also an option for that part.

Why would we use transmogrifier? There are many generic pipelines available for common cases. Flexibility to deal with different use cases. And it is actually a brilliant wayto use the iterator design pattern. Every piece of the pipeline changes a small thing, and hands the information over to the next piece.

Clients:

  • Large university site
  • High-profile government client
  • Large research institution client website

Challenge: go to Python 3, Plone 5.2 and Volto. With Volto, they spare a migration from Plone 5 to 6, at least partially. Volto is a way to sell the client a Python 3 upgrade: it is the right time to use Volto.

We use Jenkins to test the migration.

General backend things:

  • Old ATTopics to core Collections. Use collective.jsonify to export it, in a way that already looks more like a Collection.
  • Rich text: migrate to Volto blocks.
  • Post migration: collective.cover needed special handling.

What we polish to enter in Volto land:

  • We use collective folderish types.
  • deal with default pages
  • Convert RichText to Volto DraftJS
  • easily point to old website when content is not imported
  • fix some urls (we want to use resolveuid)
  • simple folders we should turn info page with listing block
  • simple collections we should turn info page with collection block

Question: can you open source the module that handles the RichText to Volto blocks migration?

Answer: yes, it is actually just 38 lines with React in Node. I tried to do it in Python, but could not get it to work. And we can sprint on it.

Open Plone Board Meeting

published Oct 25, 2019

Board Meeting at the Plone Conference 2019 in Ferrara.

See the annual report.

Jen Myers has graciously offered to stay on next year as treasurer.

The Zope Foundation has been added to the Plone Foundation. Still ongoing, complex process.

With the Pylons community, those talks have stalled at the moment. Might happen in the future.

Financially we lost more money than we usually do. There was a trademark conflict that cost a lot more than we thought.

Some money should come from the Zope Foundation by the way.

Results of the vote for the board.

48 valid votes, one invalid. 2 went to the spam folder, but we found them. 30 percent of votes were in paper.

We had a vote by the foundation membership. Voted into the Plone board have been: Victor, Erico, Chrissy, Andy, Paul, Jens, Fulvio.

Meeting closed by Paul.

Thanks to Alexander, Kim and Carol who are leaving the board.

Riccardo Lemmi: Deployment Automation

published Oct 25, 2019

Talk by Riccardo Lemmi at the Plone Conference 2019 in Ferrara.

We wanted to find an easy way to reproduce the installation process. We use:

  • Vagrant
  • Fabric
  • AWS
  • Boto 3 / awscli

Vagrant manages virtual machines and containers, for example with VirtualBox. I use the init command to do some more configuration on a default box.

Then I use Fabric to make an ssh connection to the machine and do remote actions. It uses invoke and paramiko for this. You can let it run the same actions on different machines. You can also use it to transfer files to and from the server, or use sudo to restart Apache.

More libraries with Fabric:

  • fabtools to make sure that for example Python is installed, or a user is created.
  • Cuisine: update files. You can also use this tool to ensure packages and users, so parts are very similar to fabtools.

Next as AWS, Amazon Web Services. With this we deploy production and test machines in a simple and replicable way. You can choose to add more CPUs, bigger disks, more memory, etcetara. I use EC2 (elastic compute cloud), EBS (elastic block storage) and EIP (elastic IP) for the most. Snapshots as simple backup tool. Security group rules as a firewall.

I create a machine with Boto 3 or awscli, both available with pip install. Why would I script this? To have "infrastructure as code". When you manually try to replicate a server, you can easily forget things.