I am a cloud development advocate at Microsoft.

"The Plone community is a drinking community with a software problem," quoting Alexander Loechel.

"It works on my machine..." You hear that a lot. CICD (Continuous Integration and Continuous Development) help here. I will show you that you too can be a CICD master.

You want to automate your builds, your deployments, and your testing.

You want to have change management. Sure, you use a repository for your source code. But what about database schemas and configuration?

Continuous Integration needs:

• frequest commits
• test runs for each commit
• fixing production is the main priority

Continuous Delivery, why?

• avoid a deployment risk
• real progress: a bug is fixed, and the fix is not just sitting there in a repository, but is actually live
• you get quick feedback from your users

Tasks:

• automate the build
• automate the test
• automate the deployment

For Angular you can use Angular CLI: ng build --prod --build-optimizer. Lots of flags, like --aot for ahead of time compilation, for a faster web site.

On Circle CI you can register with your github account and let it build your project, and then you can use it on for example Docker. It does require access to all your projects currently.

[Demo of Circle CI and Docker.]

There is also an Azure Container registry, where you can have private docker images. You can setup hooks that watch for a new release here, and immediately deploy the new image to production.

• code for demo: https://github.com/simonaco/circleci
• slides on slideshare

This is about storing blobs in the cloud. Mostly I will talk about storing them in S3 from Amazon, but you can store them elsewhere as well.

Earlier this year they had about 750 GB of blob data, growing by a few GB a day, and the hard disk was 90 percent full. It was also hard to keep copies of the site up to date, and keep backups. What to do?

Requirements:

• move blob data to cloud storage
• keep reasonable performance
• no changes at the app level (the Plone Site)
• smooth migration path: we did not want to bring the site down.

So should we handle it at the app level, storing an S3 url? But you need to have the image locally as well when you create scales.

Inspiration: ZODB client cache. Could we keep some blobs in there? I looked further, and Jim Fulton had already created s3blobstorage and s3blobserver. It looked experimental and not a lot of documentation, though it did not need to be a deal breaker.

So I created collective.s3blobs, which works a bit like the other two packages. When the app needs a blob, it first looks in the ZODB, then a filesystem cache, then an S3 bucket. When getting it from S3, it puts it in the filesystem cache. All recent blobs end up in the S3 blob cache, from both ZODB and the file system cache. It has a size-limited cache.

You can choose which blobs to move with an archive-blobs script, which you can pass some options. This avoided the need for migrating everything at once.

In your buildout you add a storage-wrapper with collective.s3blobs:

storage-wrapper =
  %% import collective.s3blobs
  
    cache-dir ${buildout:directory}/var/blobcache
    cache-size 100000000
    bucket-name your-s3-bucket
    %s
  

This requires a recent plone.recipe.zope2instance recipe.

It was successfull. Certainly for this site, lots of large original images are never shown, only smaller scales, so it helps a lot here.

S3 claims to offer secure storage, that you do not lose your data. We have setup rules to avoid someone accidentally deleting the blobs when they have direct access to S3, outside of Plone. We have versioning in S3.

Ideas:

• Packing needs to be implemented. The standard ZODB packing will not remove blobs from S3. I have some ideas.
• We could use CloudFront CDN to serve the images. That has security implications, although for this site it would be fine. But it costs extra, and they did not need it.

It is in stable production use, but unreleased. Needs more documentation and tests.

Code: https://github.com/collective/collective.s3blobs

Migration is just a matter of state, moving from state A to state B. The customer does not care how you get there, but you should, or it costs you too much time.

We migrated a site from Plone 4.1.3 to 5.0.6, and from a custom UI to Quaive, and from Archetypes to dexterity. The standard path is fine. But we also needed to upgrade Solr and several add-ons. And we had 10 GB of Data.fs and 300 GB of blobs, about 200K ATFiles, and previews, and versions, and about 5000 users.

For the standard path, getting the fresh data would take hours, upgrade of Plone hours, to dexterity days, and add-ons unknown. In one weekend we would need to block 5000 users. That was not really an option. So the target was less than one day.

Three environments: production on several machines, staging environment, and a migration server that was not very big.

Tip: use a migration view. You have well organised code, and have fast development with plone.reload, and you can curl it. This is better than having a script that you run with bin/zeoclient run. You get a well defined, reliable and clean upgrade path.

To rsync 300 GB of blobs takes time. Can we start a migration without blobs? Yes, by using experimental.gracefulblobsmissing. How to start:

• prepare migrating env
• copy Data.fs and Data.fs.index
• rsync blob in the background.

Some Plone upgrade bottlenecks:

1. The portal_catalog: if an index is added or needs reindexing, all objects need to be awoken and for us it took 45 minutes.
2. portal_catalog
3. portal_catalog...

Solution: go brainless. Clear the portal_catalog and run the standard Plone migration. This takes no time. For us this worked, because we wanted to touch all objects later anyway. But some upgrade steps might depend on having a fully filled catalog.

Add-ons. You may need newer versions. They may not yet be compatible. Solution: straight to the goal. We made a Plone 5.0.6 buildout with only the new code and packages that we wanted to use. So we had classes in the Data.fs that no longer existed. We used from plone.app.upgrade.utils import alias_module and used that for replacing missing classes with tiny classes that had just the few methods that were needed during migration. We cleaned up the persistent import and export steps from portal_setup before we started. We cleaned up the skins. So we focused on the result, and not on packages that would not be there anymore.

Biggest problem: @@atct_migrator for migrating from Archetypes to dexterity. This basically doubles your blobs because the old blobs are still there. It took too long for us. We needed to avoid recreating the blobs. In a migration step, for each object, we deleted it from the parent, set the newly wanted class, and added it to the parent again. Fields needed to be adapted or updated, like from DateTime to datetime. For each blob in a field, we created a new NamedBlobFile, and directly set _blob to the old blob data, so that no new blob had to be created on the file system. We never actually needed to read the blobs, so it was fine that the blobs were taking time to rsync.

Solr migration could be done in parallel, in half an hour. Then atomic updates in about two hours. SearchableText was not touched, which would have taken to long with all the parsing of text from PDF and Word.

In many cases, the standard Plone migration is fine. So if you can, just use it.

Note that we also needed to handle other catalogs, and link integrity, and some other stuff.

Philip: plone.app.contenttypes has a function for changing the class too. And functions for temporarily switching off linkintegrity.

This is an opinionated view on where we can learn, what we should stop doing, and how choices play out.

What do I mean with an omnivore's perspective? I work for a non-profit organisation. I have used Plone from version 0.99. I build sites, maintain them, renovate them, revive them, keep them, put them to sleep. So I do not build sites and then let someone else handle it. Some sites have content of over twelve years old. Some survive for three years even though the client promised me it would only be there for a year.

What systems do I use? Plone, Wordpress, Drupal, Wagtail, Mezzanine, Ghost, Pagekit, Sulu, Grav, Hugo. Those were the CMSes, well, sort of. I also use CiviVRM, Mailchimp, Odoo. These are ones that I have used in the last twelve months alone.

Some are very good. It helps that some are very new, so they have no balast.

Some good trends that I like:

• design is getting simpler and simpler
• designers got into their heads that there are mobile users
• testing is done better
• staging and production are known words for most systems

There is also the bad:

• developer centric consultantware, where for any change you need to ask a developer, instead of having a lot of power already in your hands like in Plone
• Some are so simple that it is basically brochureware. It works and looks fine as long as you have at most ten pages. So for a larger NGO: maybe not. 'Does it scale?' is really a good question here. Does it survive three years with 300 new pages a year, including several restructurings.

And of course the ugly:

• The worst ideas resurface, like the inline editing that we had in Plone.
• Once you publish it, you cannot change the url ever... Do you work in a real organisation?
• One way street, also known as data grave. Please give me a way to export the data without resorting to crawling the site.
• Security as an afterthought.

What Plone has that is really awesome and mostly not anywhere else:

• placeful content: folders with stuff in them. This means you can find it. You can move a folder somewhere else including all its stuff. Again, a flat structure is fine for ten pages, but not for more.
• collections: show a list of content somewhere else. Awesome, I want more and better of those.
• workflow: just unrivalled. Others may have a bit, but not good.
• content rules: if you do not use them, you are missing out.

Plone's happy place, where it shines, according to me:

• long term content, that you are supposed to keep for years and years. Overkill for a site that is just there for a short term action.
• skilled editors. They do not need to be full time editors, but they should be happy to at least spend half an hour to learn how the system works.
• commitment from the organisation
• where you can trust and empower your power users. They are your ambassadors.

Where can Plone approve? In the way that we approach content management as a whole. We often get bogged down in details in our daily work. But you are a captain and should act like it: Captain Janeway said 'there is coffee in that nebula, go for it,' instead of 'point the ship at direction 180.53 and warp 4.623.' Get the big picture. Tell developers (or Plone) what you want, not how exactly you want it.

Content life:

• 20 percent is spent writing the content outside of the CMS. Say Word or an email or Google docs.
• 5 percent of time is spent getting it into the CMS.
• 75 percent is spent to re-arrange it, re-use, refer, tag, archive. In other words: content management. It is annoying if it takes twenty clicks to get anything done.

Content types:

• Forget dexterity versus Archetypes for a moment.
• You have text and images.
• Embed: youtube, twitter, etc. It is there on basically every site.
• Snippets and results. Extra things like 'people you bought this article, also bought these other three, so please by them as well'.
• Office and PDF. People expect to see it this in the site. People do not see what the difference is between a Word file and an html page. Why is one opening in Word and another in the site? Make it easy for them with some integration. There are ways.
• Composite pages. This is a hard problem. We had at least seven attempts, the last one Mosaic, which is on its way, but good be better. Remember: be a captain. Make it possible to say: item A is the most important, show items B and C if you have room, drop item D and E when you are on a phone. What if we all get extremely large screens or extremely small screens in your glasses? Do you really need to go through all your pages then?

Sub sites:

• Folder, Composite and Theme.
• limited navigation
• handmade is good enough
• If the sub site is too big, with too many non standard options, make it a separate site.

WYSIWYG:

• What You See Is What You Get?
• Actually: What I See Is Most Likely Not What You Get
• Instead, Markdown is fine for most sites. It is limited, but that is a strength. The editors are less tempted to paste a Word doc into it. A preview is nice.

Stop assuming that something will appear in the left column, because on mobile there may be only one column, or it will not appear at all.

Forms:

• Basically all form frameworks are sh*t.
• PloneFormGen is Archetypes and no one is going to keep fixing the fields, so use collective.easyform.
• A quick edit drag and drop page is nice, but form creation is for power editors, so it is fine if the interface is geeky. So focus on the end user, that the end result looks good.

System setup:

• should be repeatable
• containers are here to stay, whether Docker or something else. If you want to do it in a different way, we can say in docs that you are on your own. We should stop supporting all the different deployment strategies in the world.

Configuration:

• Readability counts. It does not need to be shiny: I don't configure a site every day for hours.
• TTW: allow theming override. A different color should be possible. Allow overriding the translation strings through the web, let the marketing people do that.
• I don't want to do Javascript TTW, just on the command line with webpack or whatever. I was always in favour of TTW, but here: no.

Don't reinvent wheels. For example, one of the biggest problems in computer science: writing responsive emails. Integrate something like mosaico that has solved this.

My practical wishlist for users:

• TinyMCE: use Markdown and raw
• tiles and layout recipes
• Make 'embed' easy.

Give me more:

• smart media handling, without letting me handle it manually
• consistency in UI
• clevererer collections: can we have something that says: these three items are relevant for you?

Team player:

• Good import and export should be in core. JSON dump is fine.
• REST and GraphQL: good if we can have both.
• Office integration when needed for intranet sites
• although security is hard

System:

• For configuration can we move from XML to YAML? No, not JSON, as that is unreadable and unwritable too.
• One Plone per Zope would be much easier.
• A command line interface to do basic maintenance would be great.
• roundtrip configuration: export config and import in another container. We sort of have this, but it is hard to find.
• For power users, function is more important than form.
• Warn me when updates are needed. Should be configurable. Probably not web downloadable updates, for security reasons.

Make it so!

Maybe you totally disagree, that is fine. Come talk to me and we can start a discussion. I am deeply partial to Plone and let's make it better.