Encolpe Degoute talks about ways to investigate for the future of Plone, during the Plone conference 2012. Let's look at Plone on various levels, from low to high, and see what can be changed.

Plone uses Python and C code.

What is the next Python to use? Options could be:

• CPython 2.7
• CPython 3.3
• Pypy
• Stackless Python

I think we should head for CPython 3.3 or Stackless Python.

Pypy does not (currently) use distutils, which is a big problem for Plone.

Does someone want to remove the GIL, Global Interpreter Lock? It would be good for performance, allowing you to use multiple CPUs, but it is hard to do.

For the network layer, Twisted is one of the worst frameworks to use. But it is written in Zope. So we should improve that, or switch. Synchronous versus Asynchronous. In Django you can switch the code that handles the network layer, for example Gunicorn. It is a big work, to be done in Zope 4 or Bluebream.

For storage we use the ZODB, storing objects as Python pickles. It uses a simple file (Data.fs) or a database (using RelStorage). A problem is that all transactions are stored in the ZODB, also when they are not useful anymore, which is why you need to pack the ZODB regularly.

Why Pickle? It is a Python specific format that can store binary objects. For more speed you need cPickle. If you want to replace it, you could use:

• JSON. It lacks schema validation. There is some variation in binary format.
• XML. You can use this to exchange messages, but it can be used as a storage format too. That should fit all the needs.

What can we improve in the ZODB?

• For B-Trees we can use stratified B-Trees, which is interesting for bigger sites.
• Better catalog indexes: use relational database power with RelStorage. The main developer of RelStorage does not want this in the core, it should be an extra package.
• Pack the ZODB: have a sub process that automatically cleans up old transactions.

Can we replace the ZODB? There are some production ready alternative databases, but they cannot completely replace the ZODB:

• Cassandra
• MongoDB

What about the basic framework, what can we improve there?

Have you looked with ZopeProfiler at a Plone site? Lots of things are called lots of times.

For a function of five lines, the initializing of the function normally takes more times than the execution. So we may want to use less functions. We need to use more Zope functions directly.

We need more asynchronouw services in Plone:

• Mail service
• transaction cleanup
• REST api

Can we switch to Bluebream and the ZTK instead of Zope 2? Are we ready for Zope 4?

For Plone, should we keep the CMF or push it in the ZTK? It was agreed that it was useful, but nobody wanted to do the work.

Which visual editor should we use? There are several options:

• TinyMCE
• FCKeditor
• Aloah
• VIE and Create.js

Main problem of all of them: integration.

In Plone we can use dexterity. We could switch everything to this instead of Archetypes. There is no toolkit currently to migrate content types and content objects from Archetypes to dexterity. That would be good to work on.

Have som file format integration. Plone should be able to create a documents from its internal format, like Open Document and PDF.

What about semantics? For example Schema.org. How can we include microformats in a Plone site? Did somebody work with Virtuoso?

So: lots of things that we could work on!

Asko Soukka and Jukka Ojaniemi talk about Plone, RabbitMQ and messaging that just works, during the Plone conference 2012.

Why message queues? We needed to run asynchronous tasks. Difficult to do well within the ZODB, once you are talking about a decent amount of requests. So we use something external.

AMQP is like a postbox: Advanced Message Queue Protocol.

• A message consists of a label and payload.
• Producer is a program which sends messages to exchange.
• Consumer is a program which waits to receive the messages.
• Exchange sits between producer and consumer
• Bindings are routes from exchange to a queue.
• Queues store messages.

You can build several messaging systems, like publication/subscription and RPC (client to server and then back to client).

RabbitMQ implements AMQP. Written in Erlang. Industry tested and reliable.

Sounds difficult? Well, messaging in Plone can be easy.

Example with collective.zamqp for publish and subscribe announcement. Get a utility, register a producer and publish a message in basically three lines. Handling incoming messages: register an event handler for IMessageArrivedEvent.

Beaker comes with an asynchronous connection utility. Zope uses a similar technique under the hood, which handles transactions for us. To use collective.zamqp you need to define some additional zope configuration in your buildout.

Code and demos: https://github.com/datakurre/collective.zamqp https://github.com/datakurre/collective.zamqpdemos https://github.com/datakurre/chatbehavior

Other relevant links: https://www.rabbitmq.com/getstarted.html https://www.amqp.org/about/examples

Slides of this talk: http://goo.gl/RDYZc

Matt Hamilton, Matt Sital-Singh talk about transmogrifier, the good, the bad and the ugly, during the Plone conference 2012.

This is about content migration. A quick show of hands: most people in the room have heard of transmogrifier. About half have used it in some way.

`collective.transmogrifier is the basis, with some companion tools with recipes or extras, like plone.app.transmogrifier and quintagroup.transmogrifier. Simple cases are very well documented. There have been many talks about it, so Google for it, for example Lennarts Regebro during Plone conference 2009, and Clayton Parker.

All the talks are mostly about how easy it is. We are going to talk about the rest as well, some things that you should be aware off.

Content migration in itself is hard. 95 percent is straight forward, but then there are the corner cases which are hard. You may have to hold to a short maintenance window. There always some changes that people want next to simply importing the old content. You may think you have everything covered but seven folders deep some item is not exported for some reason and you only find out too late. The ZODB is a bit harder for that than a simple relational database.

There are a lot of recipes, blue prints of how to do the simple cases. Sometimes you only have one shot. For one client we had a weekend to migrate a large 2.5 Plone Site to 4. You should be prepared, and have some test runs before that. You need to know how long the migration takes and can mostly not afford mistakes.

We have some code examples that probably should be moved to some of the upstream packages: https://github.com/netsight/netsight.transmogrifier

One of the things I tried to do, was divide the mgiration up into more sensible amounts. So divide a site into useful sections, minimise cross references and cleanup later. It could be as simple as first importing the users, then importing the content.

Killing the catalog is a good option. As you go along, Plone is merrily indexing and reindexing stuff. Cataloging is a large part of the time. So you could try out switching the catalog off and on again in the transmogrify pipeline.

Watch out for some corners, like that the modification date of all items will be set to today when you are migrating.

Watch out for running out of memory. You may need to restart Zope halfway through so you free up memory, depending on which versions you are using.

A migration can be a good time to get rid of some custom content types, when you have created some new types where in practice the standard types are actually good enough.

There are some dark corners. quintagroup.transmogrifier uses Marshall under the hood. This seemed to be okay, but we ran into some xml problems within Marshall, where the export format was different from the import format. Marshall was last touched in 2006, so it is hard to figure out if there was a reason for this. We have fixed some bugs there.

In netsight.transmogrifier we made some more dynamic pipelines, so look at those.

Takeaway: transmogrifier works. Some glue code needs to be published.

Steve: look at collective.jsonify instead of Marshall. Also: transmogrifying from Archetypes to dexterity works too.

How do you verify that it works? We do manual inspection. You could use web spiders to check things.

Some things do not come over, like the portal_redirection tool and portal_archivist with older revisions.

This is a talk from the Plone conference 2012.

Matt Hamilton and Eric Steele hold the keynote speach on the State of Plone.

Matt Hamilton is the current President of the Plone Foundation. Eric Steele is the Plone 4 release manager.

So, what has happened the last twelve months? Sometimes you think nothing has happened, but when you start writing it down you get really excited because so much is actually going on. This year a Plone Konferenz in Munich, Plone Symposium South America, Plone Open Garden in Sorrento, Plone Symposium East in the US. So several regional conferences in local languages, which is fantastic. In total between five and six hundred attendees. These are grass roots events, not coordinated by the Plone Foundation, which is good. At this Plone Conference, there are people from 27 different countries. We can be proud of our global diversity.

We have had the Plone 4.2 release and 4.3 is now in alpha. A lot is done during sprints. We average one major Plone sprint per month currently. 5597 commits in the past year, 208 contributors. The past month saw 66 people committing to the core of Plone, with 13 of them being new. We will probably break that record this month. We loose some people who move on to new projects, but there are still more people coming in than have left. The Plone core development list is very active.

There are fourteen new members in the Plone Foundation. These are members who have made major contributions to Plone.

World Plone Day had lots of activities worldwide. We can use more events and developers and users in North Africa, Middle East, India.

We had our sixtieth Plone TuneUp online event, shepherded by Carol Ganz from Six Feet Up. Lots of tickets have been closed. In just that last day, 34 tickets have been solved.

We have the Plone Roadmap. The roadmap team created this ongoing document, which is a high level summary of goals of what we want to do with Plone. Some goals may shift. We want to iterate, have more frequent releases. Introduce new technologies early, before they become the standard. We want to be more aggressive about moving under-used and under-maintained features out of core. We want to keep sane upgrades, instead of possibly breaking backwards compatibility in Plone 5.

Plone has moved to github. All of Plone core has moved, making merges, patches, pull requests much easier. Much of the collective has moved too.

There is a new Windows Installer, which is often the first impression a new user gets. Enfold has done a lot of work here. Ross Patterson has been working on getting a much more native installer. The installer uses the same base buildout as other installers. We have been looking at managed cloud deployment options. We have Bitnami, Turnkey, Ploud.

Elizabeth Leddy and Karl Johann Kleist have been cleaning up the Plone Trac, the ticket system, cleaning up tags, reassigned bugs. Fantastic, really helpful. We have made the bug report form simpler. A QA team triages tickets, checking if they are valid and clear.

Timo Stollenwerk leads an effort on automated testing. This is about maintaining code quality. Jenkins is a big part of this. Recently Travis-CI. We are starting to use the Robot Framework, which helps for testing javascript, which is very much needed with the amount of javascript code we have.

This is a talk from the Plone conference 2012.

Jan Jongboom from Cloud9 talks about Javascript.

In 1995 Livescript appeared, developed by Mozilla. A simple scripting language. They called it Javascript, though it was not really like Java at all. It was more like Scheme.

In 1999 Microsoft wanted to build a web version of Outlook 2000. At that point it was not possible to send data from the client to the server without refreshing your whole page. So the XMLHttpRequest was born. This means you can have asynchronous applications.

So Google wanted to build something cool as well. So they launched Gmail in 2004.

Javascript did not get a lot of love in Plone in the beginning. A problem was that there were lots of difference between the Javascript implementation of browsers. The book 'Javascript, the good parts' is eight times smaller than the definitive Guide to Java.

In 2004 (?) JQuery got launched. Maybe the most popular library ever. The power comes from Sizzle, which allows you to easily select DOM elements on your page.

From about 2010 Javascript got more popular than Java, according to some definitions. People started to care. VMs got faster and embeddable. Ecmascript 5 got introduced.

In 2009 the JSConf was the first conference completely about Javascript. NodeJS was introduced at that conference. It allows you to use Javascript on the server. You could think: "Ah, nice, Google's V8 engine as an executable." But it is more powerful than that. LibUV is in there. It abstracts hardware differences away. You would want code to work asynchronously, and operating systems do not normally work that way: you get a file and once you have the data you continue. With LibUV you get events on files, networking, DNS, etcetera: once data starts flowing in, your method gets called so it can continue its work.

Threads can easily get blocked by IO operations. You have a powerful server but due to this, you may still be limited to 300 concurrent connections (if your thread uses ten MB of memory and you have 3GB of memory), or 300 users logged in to the system. With non-blocking threads you can handle tens of thousands of connections.

nodeJS is the Javascript bindings to LibUV. You use standard libraries in Javascript. This Javascript code is then executed in the V8 engine.

Couldn't you use Twisted, the asynchronous Python framework for this? Why would you choose Javascript over Lua or Python or C++? Most Python code is not build to be used asynchronously. In Javascript it is normal.

So we decided to build Cloud9. For most programming languages there are debugging tools and good syntax highlighting. For Javascript: not so much. Cloud9 is an online tool that can do that. You can download it and install it on your server. We give you a free Linux VM with a real terminal. The coolest feature is collaboration. See your colleague type. Debug together.

In our data center we only have Cloud9 code, our IDE server. In another part of the data center we have the remote VMs. We ship it with Python 2.6.6, but you can compile 2.7 or 3.3 on it. You can run MongoDB or Postgres on it, or whatever you want, like Plone. I had problems with it, but you should be fine.

If the standard VM does not have enough power (OpenShift, 128/512 MB Ram, limited resources), you can bring your own VM. I tried the Plone UnifiedInstaller on an Amazon ec2 machine, which I will demonstrate now, if the wireless connection is working good enough. Otherwise a video will have to do. You will see a collaborative session in two browsers where a Plone instance is started and I get a co-worker to assist me on that same machine in his own browser for editing the logo of the Plone instance.

The Cloud9 IDE gives you code completion in the Ace editor, which makes it easier to edit Javascript (like nodeJS) and in a few months Python should be better.

The code is on github so go ahead and fork it.

See http://c9.io/

We use Concorde for the peer to peer messages. Unfortunately closed source.

Could it be used for online sprints, working on some github code of Plone together, assisting someone on the other side of the world? Yes. For open source it is free to use, for closed source you have one free project, and there are paid plans.

There are a couple of issues with the keyboard in the browser, so if you want to use some editor on the command line. You can override some keys in your browser.