Symbol of generosity for us in the United States is Santa Claus. He distributes gifts for free in the evening before Christmas. Now there is something called SantaCon, where young men go around dressed as Santa and get drunk.

I saw a young Santa who was asked for money by a homeless man in a wheel chair. He gave five dollar. The homeless man did not say thank you. Maybe he was just tired. But then the Santa got enraged, demanding that the man thank him. So here we see power in action. Not a pretty sight. How does that happen.

A new colleague of mine at MongoDB asked me about something. I said it was going to be in the next release, and thought that would finish the conversation. Then he asked me to make more effort: what is the ticket number. For some reason this angered me, and I answered: 'I memorised all ticket numbers, it is 12345.' Then he came back at me: 'Are you sure? I cannot find that ticket.' I regretted my answer.

It is not okay to get mad and lash out at people because they make us angry.

On StackOverflow Python is projected to become the most popular language next year. I think one of the reasons is that we are an inclusive community. We try to be nice to new users. But we are not as welcoming as we could be. I searched for rude answers on StackOverflow and it was not hard to find.

• Question: 'How to I convert GIF to PNG in Python?'
• Answer: 'Just Google it, you are lazy.'

That is not a good answer. It hurts our community, because it drives people out.

Why do we do this? There cam be several reasons:

• We are generous, but get no reward, like the bad Santa.
• Unexpected difficulty: I give an answer, and then the person does not understand me and asks a followup question, and suddenly I am facing more trouble than I want.
• Sense of obligation. We sense we need to give some kind of answer, so we give a rude one.

If I get a promotion, or delete ten emails, it gives temporary satisfaction, but does not really make me more satisfied in the long run. I got a lazy (in my eyes) comment on a Python video that took long to make, and I got angry. I have learned to not make a rude remark. Instead I closed the browser tab. That is the easy way.

When faced with a question, ask yourself: am I angry?

• Yes, I am angry. Walk away. If you can't say anything nice, don't say anything at all.
• Yes, I am angry, but what is my intention? My job may be to answer questions. I may want to take good care of my code, or nurture the next generation of programmers. Thinking about that, am I still angry?
• No. Start your answer with: 'Good question!' Even if it was not really a good question at all. It makes you and the other feel good.
• Is it complete? You are done.
• No, the person asks a followup question. Go back to the first step: am I angry?

Take the 3-breath vow: "I vow to take 3 breaths before I answer a question online."

I am a cloud development advocate at Microsoft.

"The Plone community is a drinking community with a software problem," quoting Alexander Loechel.

"It works on my machine..." You hear that a lot. CICD (Continuous Integration and Continuous Development) help here. I will show you that you too can be a CICD master.

You want to automate your builds, your deployments, and your testing.

You want to have change management. Sure, you use a repository for your source code. But what about database schemas and configuration?

Continuous Integration needs:

• frequest commits
• test runs for each commit
• fixing production is the main priority

Continuous Delivery, why?

• avoid a deployment risk
• real progress: a bug is fixed, and the fix is not just sitting there in a repository, but is actually live
• you get quick feedback from your users

Tasks:

• automate the build
• automate the test
• automate the deployment

For Angular you can use Angular CLI: ng build --prod --build-optimizer. Lots of flags, like --aot for ahead of time compilation, for a faster web site.

On Circle CI you can register with your github account and let it build your project, and then you can use it on for example Docker. It does require access to all your projects currently.

[Demo of Circle CI and Docker.]

There is also an Azure Container registry, where you can have private docker images. You can setup hooks that watch for a new release here, and immediately deploy the new image to production.

• code for demo: https://github.com/simonaco/circleci
• slides on slideshare

This is about storing blobs in the cloud. Mostly I will talk about storing them in S3 from Amazon, but you can store them elsewhere as well.

Earlier this year they had about 750 GB of blob data, growing by a few GB a day, and the hard disk was 90 percent full. It was also hard to keep copies of the site up to date, and keep backups. What to do?

Requirements:

• move blob data to cloud storage
• keep reasonable performance
• no changes at the app level (the Plone Site)
• smooth migration path: we did not want to bring the site down.

So should we handle it at the app level, storing an S3 url? But you need to have the image locally as well when you create scales.

Inspiration: ZODB client cache. Could we keep some blobs in there? I looked further, and Jim Fulton had already created s3blobstorage and s3blobserver. It looked experimental and not a lot of documentation, though it did not need to be a deal breaker.

So I created collective.s3blobs, which works a bit like the other two packages. When the app needs a blob, it first looks in the ZODB, then a filesystem cache, then an S3 bucket. When getting it from S3, it puts it in the filesystem cache. All recent blobs end up in the S3 blob cache, from both ZODB and the file system cache. It has a size-limited cache.

You can choose which blobs to move with an archive-blobs script, which you can pass some options. This avoided the need for migrating everything at once.

In your buildout you add a storage-wrapper with collective.s3blobs:

storage-wrapper =
  %% import collective.s3blobs
  
    cache-dir ${buildout:directory}/var/blobcache
    cache-size 100000000
    bucket-name your-s3-bucket
    %s
  

This requires a recent plone.recipe.zope2instance recipe.

It was successfull. Certainly for this site, lots of large original images are never shown, only smaller scales, so it helps a lot here.

S3 claims to offer secure storage, that you do not lose your data. We have setup rules to avoid someone accidentally deleting the blobs when they have direct access to S3, outside of Plone. We have versioning in S3.

Ideas:

• Packing needs to be implemented. The standard ZODB packing will not remove blobs from S3. I have some ideas.
• We could use CloudFront CDN to serve the images. That has security implications, although for this site it would be fine. But it costs extra, and they did not need it.

It is in stable production use, but unreleased. Needs more documentation and tests.

Code: https://github.com/collective/collective.s3blobs

Migration is just a matter of state, moving from state A to state B. The customer does not care how you get there, but you should, or it costs you too much time.

We migrated a site from Plone 4.1.3 to 5.0.6, and from a custom UI to Quaive, and from Archetypes to dexterity. The standard path is fine. But we also needed to upgrade Solr and several add-ons. And we had 10 GB of Data.fs and 300 GB of blobs, about 200K ATFiles, and previews, and versions, and about 5000 users.

For the standard path, getting the fresh data would take hours, upgrade of Plone hours, to dexterity days, and add-ons unknown. In one weekend we would need to block 5000 users. That was not really an option. So the target was less than one day.

Three environments: production on several machines, staging environment, and a migration server that was not very big.

Tip: use a migration view. You have well organised code, and have fast development with plone.reload, and you can curl it. This is better than having a script that you run with bin/zeoclient run. You get a well defined, reliable and clean upgrade path.

To rsync 300 GB of blobs takes time. Can we start a migration without blobs? Yes, by using experimental.gracefulblobsmissing. How to start:

• prepare migrating env
• copy Data.fs and Data.fs.index
• rsync blob in the background.

Some Plone upgrade bottlenecks:

1. The portal_catalog: if an index is added or needs reindexing, all objects need to be awoken and for us it took 45 minutes.
2. portal_catalog
3. portal_catalog...

Solution: go brainless. Clear the portal_catalog and run the standard Plone migration. This takes no time. For us this worked, because we wanted to touch all objects later anyway. But some upgrade steps might depend on having a fully filled catalog.

Add-ons. You may need newer versions. They may not yet be compatible. Solution: straight to the goal. We made a Plone 5.0.6 buildout with only the new code and packages that we wanted to use. So we had classes in the Data.fs that no longer existed. We used from plone.app.upgrade.utils import alias_module and used that for replacing missing classes with tiny classes that had just the few methods that were needed during migration. We cleaned up the persistent import and export steps from portal_setup before we started. We cleaned up the skins. So we focused on the result, and not on packages that would not be there anymore.

Biggest problem: @@atct_migrator for migrating from Archetypes to dexterity. This basically doubles your blobs because the old blobs are still there. It took too long for us. We needed to avoid recreating the blobs. In a migration step, for each object, we deleted it from the parent, set the newly wanted class, and added it to the parent again. Fields needed to be adapted or updated, like from DateTime to datetime. For each blob in a field, we created a new NamedBlobFile, and directly set _blob to the old blob data, so that no new blob had to be created on the file system. We never actually needed to read the blobs, so it was fine that the blobs were taking time to rsync.

Solr migration could be done in parallel, in half an hour. Then atomic updates in about two hours. SearchableText was not touched, which would have taken to long with all the parsing of text from PDF and Word.

In many cases, the standard Plone migration is fine. So if you can, just use it.

Note that we also needed to handle other catalogs, and link integrity, and some other stuff.

Philip: plone.app.contenttypes has a function for changing the class too. And functions for temporarily switching off linkintegrity.