PLOG Wednesday noon: backend

Timo Stollenwerk leads discussion about status of our current backend stack.

Gather a full report, what happens to the components we are using, put it together. Start discussion about what we can do about it.

[Note that this is not necessarily the vision of the Plone community. This is a gathering of thoughts of individuals present at the meeting. Not everyone will agree.]

• Python 2 and 3. Python 3 is pretty far away with our current stack. Several packages work on Python 3, even some of the 'scary' parts. We would need to solve other problems first.
• Zope 2 currently has no maintainer. Oh, Godefroid is release manager. People are still contributing, but not everything gets released. There are automated builds for testing. If we need something from Zope 2, we can talk to people, but we need to develop it ourselves. We use the latest Zope 2 release.
• Zope toolkit. On github. Version wise we are behind, which is not actually that bad. It is quite stable. Version updates seem to have mostly been for some reorganisation, and also getting rid of zope.app stuff. Various people still has commit rights, people do stuff when they need it. Tres Seaver and Marius Gedminas change stuff, avoiding bitrot and keeping tests running. No big plan. The Zope Foundation organisation is not really active.
• ZODB 3. We could move to 4. A Python 2 ZODB cannot be migrated to Python 3. But the package supports both.
• ZCatalog. David Glick and Hanno Schlichting worked on it. It needs Zope 2, so no one outside is using it. Technology is way beyond its limits, holding catalog data in zodb is over. ZCatalog is what makes ZODB unmarketable, becoming a bottle neck.
• CMF. Still on subversion on zope.org, which makes our continuous integration a bit unstable. Basically only one committer, who prefers not to switch to github. There are some ways to automatically sync subversion to github. We can talk to Tres Seaver as well. In CMF everything depends on everything, hard to separate. We are subclassing much of CMF. Essential is at least the portal_types tool and workflow. We are not actually using the latest release I think, which might remove the CMFDefault dependency, not sure. Custom code out there will reference core CMF stuff that is not really in the API, like underscore methods. We can at least try to rip out stuff and see what happens. Flatten the inheritance structure. Create an explorative team that takes a few hours to try things out and come up with an estimate.
• Archetypes and dexterity content types. Still supporting Archetypes, kind of, mostly some UI issues. Testing against both Archetypes and dexterity content types is tricky, brittle. If people use it and depend on it, it will be maintained. Suggest deprecating it in Plone 5.1, rip it out in 6? If people put effort in it, it is fine. If it is moved out of the core, it is dying. There is migration code for data. Plone community is smart, but we cannot handle all parts forever, we might need to drop stuff that is hardly getting used. If we want to rip it out in Plone 6, we should announce that officially. We need a roadmap for that. And find somebody who cares about Archetypes enough to keep an eye on it, that it remains working nicely. Please step up if you care, maybe we could still keep it in Plone then. Making it more smooth to move content types from Archetypes to dexterity, also in the actual Python code instead of just the data, would be very helpful. Make a script that can take an Archetypes schema and creates a dexterity schema from it, as a start. We are good at creating ideas, but bad at communicating what people would need to do, that they should move out of Archetypes. Publish it on plone.org, etcetera. Make a decision and communicate.
• Zope 4. Many options for the next step that we could take. It matters a lot where you want to go. ZODB and object model make Plone unique, flexible, very good for content management. So on long term: keep ZODB, do not move to relational database (well, RelStorage is fine). Storing json in postgres could be an option too. Writing object oriented, polymorphic code is sane with ZODB, including hierarchical data. Security on object level is awesome, not for only View permission, but other permissions, which other frameworks don't do. RestrictedPython that checks whether you have the right permissions is very nice. Some don't agree. Role based access control is great. Building products on top of Plone is very powerful. A lot of that is based on features in CMF, think of workflows, etc. But it would be good to get rid of all the layers on layers on layers. There is so much Plone. You currently need to be an expert developer to cope with this. Plone has been around for a long, long time, which is actually good. We have lots of knowledge, we need some new answers on which parts could and should really be removed, without irreparably throwing stuff away. Pyramid has the same concepts, making it very familiar, clean, without over designing things, with expressive APIs. It is not about whether we should use Pyramid, but about figuring out what the essence is that we should keep. And what small steps can be done now to make changes in the future easier. For Pyramid the right solution was to rip everything out and build the essence. This may not be the best approach for Plone. We need to stop building layers and layers and layers. We need a stronger core with the essence. The DemoStorage is really very helpful for testing, much harder to do right in relational databases. A cleanup is not very rewarding to those working on it, you need to know where you want to go, where you want to end up. Rip out stuff and move to the shiny stuff on the front-end. Think about how to cut away significant parts of the complexity. If Plone is reduced to a json store, should we want that, we can rip out much. If you know where you want to be in five years, it avoids the temptation to do intermediary stuff that you will rip out a little while later. Complex systems evolved from simple systems that already work. We cannot start from scratch. Plone has always been an innovator. Keep doing that. But also be smart to switch to the best of breed if another community has come up with something better.
• Move the discussion back to Zope 4. Several years ago Laurence did work, some lost during a move to github. It is a significant amount of work. Does it really get us where we want to be? Do we need to rebuild everything we currently have in Zope 4, or do we need to build something else, sidestepping the current code? If we have a RESTful API, getting a shiny front-end becomes easier. Makes mobile development much easier too. All web development frameworks face this problem. A full JSON api will not be there overnight. It will be hybrid. We need to keep working on our Javascript story. Pyramid has template renderers and JSON renderers, making for a clearer separation of data and presentation. We can just serve the JSON and have some more or less separate front-end show it. Plone is a CMS, not really a framework, so it differs a lot from Pyramid. Pyramid is more like Zope. In Pyramid you build your application from the inside out, with Pyramid just as a tool.

So likely directions from here:

• Reduce CMF to one layer within Plone.
• RESTful JSON api. This has the possibility of making a lot of the code irrelevant, so making it easier to remove. Makes it easier to use a different layer below for publishing.

There is no roadmap in the Javascript world, it can move everywhere, so we cannot know exactly what we would need to support or what we can count on there.

Do not forget the Python API in plone.api. Add-ons and higher level Plone layers should use this API. Still hard to get rid of the underlying stuff.

What you want in a JSON API is not a one on one rebuild of the Python API. More content centric: a JSON representation of the content.

More important to focus on where we want to be than on how we solve it. A vision is needed as a start. And more important to have a vision than to have the correct vision. Otherwise you go in circles. Do not plan on too many things that might possibly go wrong. Be bold. Do not loose vision based on what your customers want, which customers you may loose.