We wanted to find an easy way to reproduce the installation process. We use:

• Vagrant
• Fabric
• AWS
• Boto 3 / awscli

Vagrant manages virtual machines and containers, for example with VirtualBox. I use the init command to do some more configuration on a default box.

Then I use Fabric to make an ssh connection to the machine and do remote actions. It uses invoke and paramiko for this. You can let it run the same actions on different machines. You can also use it to transfer files to and from the server, or use sudo to restart Apache.

More libraries with Fabric:

• fabtools to make sure that for example Python is installed, or a user is created.
• Cuisine: update files. You can also use this tool to ensure packages and users, so parts are very similar to fabtools.

Next as AWS, Amazon Web Services. With this we deploy production and test machines in a simple and replicable way. You can choose to add more CPUs, bigger disks, more memory, etcetara. I use EC2 (elastic compute cloud), EBS (elastic block storage) and EIP (elastic IP) for the most. Snapshots as simple backup tool. Security group rules as a firewall.

I create a machine with Boto 3 or awscli, both available with pip install. Why would I script this? To have "infrastructure as code". When you manually try to replicate a server, you can easily forget things.

I have some upgrade code that you can use, or for some parts copy and adapt for your use case.

Code: https://github.com/collective/collective.migrationhelpers

Upgrade steps:

• Do imports in the function.
• Do them conditional, so it does not fail if some package does not exist.
• Do not let a step fail when run a second time.

You may want to disable LDAP temporarily in an upgrade step.

Get the birds eye view, with some code that reports:

• how many items are there
• which portal types
• how big
• local roles
• etcetera
• How many items are there that need to be replaced or removed, like PloneFormGen or Collage.

Divide and conquer:

• Deal with one problem at a time.
• Ignore problems that don't block you. You may try to solve something in the old Plone 4.3 site which is already fixed just by completing the migration to Plone 5.

You can register upgrade steps conditionally in zcml if needed, for example with zcml:condition="installed plone-52".

Make big problems small:

• Write something that removes 98 percent of your content, for testing. Keep the structure in place though: Folders may have portlets or local roles that give problems that you want to know of.
• Do not migrate blobs. The PDF that lives on the filesystem will not change. Move the blobstorage out of the way, use experimental.gracefulblobmissing during migration, and move blobstorage in again.
• Copy the Data.fs.index too, if it is a really big site.

Forget the past:

• remove all revisions
• maybe manually use collective.revisionmanager for this.
• pack the database with zero days (bin/zeopack keeps the last day by default).
• Remove no longer needed portlets.
• You can use uninstall and upgrade profiles. I like having the upgrade code in Python, but sometimes a profile is much easier.
• Remove utilities and adapters from add-ons that will be removed.
• Sometimes you cannot easily remove a package. You can make an alias for it. plone.app.upgrade has code for this, for example to not crash on an old site that still expects the kupu editor package somewhere.

Migrating LinguaPlone:

• Content editors may have done crazy things, combining folders and content from different languages.
• We have code to migrate this to plone.app.multilingual in migration.plonehelpers.

Update to Plone 5.2:

• Use Python 2.7 at first.
• Include Archetypes if needed for migration.
• Run the migration to migrate to dexterity.

Archetypes to dexterity:

• Use the methods from pac_migration from plone.app.contenttypes.
• Start with the containers/folders.
• Do one type at a time.
• Especially for blob-like items there are options to speed this up, like disabling updating SearcheableText.
• You can write custom migrators, which might just need a few lines.

Alternative: inplace migrator from ftw.upgrade. Interesting for large folders. Might be a nice PR to get this part into core.

To Python 3:

• We need to remove some archetypes tools.
• Then remove the Archetypes eggs from buildout and use Python 3.
• Run the zodbupdate script.
• When using RelStorage, you may want to switch to filestorage temporarily.
• Read the documentation please, especially: - upgrade to Python 3 - upgrade zodb to Python 3

Why do it this way? I don't want every Plone company to have their own migration code stashed away somewhere. Please contribute to the core.

Upgrade to Plone 6? Install plone.restapi, use Volto, done.

A year ago, ZServer was one of the last Zope parts that did not run on Python 3. The idea was to use WSGI instead. I wondered why Python 3 would not be possible. So I tried porting it. It worked. And I added websocket support. You can use this to automatically push pages from Plone to Gatsby. Or show a notification when a page needs review, via content rules.

I replaced the old Medusa server with Twisted. ZServer was optionally using Twisted years ago, but this was ripped out, and I don't know why. But I got it working again. So for HTTP and webdav we have a Twisted http server and thread pool with an async FileSender, an AsyncIO/uvloop when avilable, standard ZConfig configuration and logging, And HTTP/2 works as well.

Bonus: websockets with PubSub messaging. It uses the Autobahn Twisted WebSocket protocol. ZeroMQ PubSub cockets using IPC-socket. The connection autosubscribes to local events. The server publishes a guarded event, for example indicating that an event is restricted to Editors.

One problem at this moment with the wsgi setup using waitress, is that the requests are limited to the number of wsgi threads, so server large files can become problematic. There are probably ways around this, but ZServer does not have this problem.

Status:

• My ZServer fork is at https://github.com/datakurre/ZServer. See mostly branch datakurre/master.
• You will need a special branch of plone.recipe.zope2instance as well.
• Plus collective.wsevents, plonectl, collective.taskqueue.

Upstream:

• In Zope 4, WebDav was made dependent on ZServer, by mistake?
• Zope 5 assumes ZServer and WebDav no longer exist.

Remaining steps:

• Get it upstream or not?
• Eliminate dead code.
• QA: tests, documentation
• Release it.
• Restore wsgi=off on Python 3.

I would be interested to hear when others are interested in using this. It is better if more people use this.

I am an introvert from Finland, so I did not ask the Zope people, or visited the Zope sprints.

Jens: There was no real reason to drop ZServer, except that no one has tried it because the code was so very old. Now you got it working, let's get this into Zope.

See the slides.

It is hard to see what part of machine learning is over hyped, and what part is actually useful.

Basis of ML (Machine Learning) is to get input and transform it to output. The ML gets a picture of a duck, and it gives as answer "duck". That is image recognition. You train the ML with images that you already label. And then you give it an image, and hope it gives a good answer. Same: from English to Japanese text. From an fMRI scan to an image visualized.

Simpler example: solar panel. Input: time of day, output: how much generated power. You would train this with data. The ML would turn this into a function that gives an approximately good answer.

Simplest model: linear regression. Turn the data into a function like this:

a * X + b

No line through the training points will be perfect. You find the line that minimizes the average error. So the ML uses linear regression to find a and b.

In our example, we try to guess the amount of mojitos we sell, based on the number of passengers that a daily boat brings to our beach bar.

But this may also depend on the temperature, or the number of sharks. With two variables you would not have a line, but a plane. With n, you would get an n-dimensional shape. You have n inputs, and give a weight to each input, and add them.

This is about numbers. For image recognition, you don't get a number as output. But we can apply a function to the result, and get a number between 0 and 1 that gives a likelyhood. For an image we may have 0.02 certainty that it is a cat, and 0.9 that it is a duck, so our answer will be the highest: a duck. Translated to the mojitos: what is the likelyhood that my bar breaks even?

This system with weights and a reducer function is called a perceptron. With a short Python program I got 90 percent accuracy on the standard NDIST test. We can do better.

We look at neural networks. This is basically: mash two perceptrons together. Or more. We are finding a function that approximates the data, and reduce the error iteratively.

What would I say is deep learning? Why has it grown?

1. Neural networks with many layers...
2. ... trained with a lot of data...
3. ... with specialized architectures.

We helped Facebook with neural networks by tagging our photos. We gave them training data!

A lot of engineering is going into deep learning.

Generative Adversarial Networks: GANs. Example: a horse discriminator. Train the system with images of horses and others, and it should answer with: yes or no horse.

Other part: horse generator. Randomly generate images for feeding to the horse discriminator. You train this by saying: you did or did not manage to trick the discriminator. And you try to get better. I trained this a night, and after about a million iterations, I got pictures that are not quite horses, there is something wrong, but they are amazingly close.