Matt Hamilton, Matt Sital-Singh talk about transmogrifier, the good, the bad and the ugly, during the Plone conference 2012.

This is about content migration. A quick show of hands: most people in the room have heard of transmogrifier. About half have used it in some way.

`collective.transmogrifier is the basis, with some companion tools with recipes or extras, like plone.app.transmogrifier and quintagroup.transmogrifier. Simple cases are very well documented. There have been many talks about it, so Google for it, for example Lennarts Regebro during Plone conference 2009, and Clayton Parker.

All the talks are mostly about how easy it is. We are going to talk about the rest as well, some things that you should be aware off.

Content migration in itself is hard. 95 percent is straight forward, but then there are the corner cases which are hard. You may have to hold to a short maintenance window. There always some changes that people want next to simply importing the old content. You may think you have everything covered but seven folders deep some item is not exported for some reason and you only find out too late. The ZODB is a bit harder for that than a simple relational database.

There are a lot of recipes, blue prints of how to do the simple cases. Sometimes you only have one shot. For one client we had a weekend to migrate a large 2.5 Plone Site to 4. You should be prepared, and have some test runs before that. You need to know how long the migration takes and can mostly not afford mistakes.

We have some code examples that probably should be moved to some of the upstream packages: https://github.com/netsight/netsight.transmogrifier

One of the things I tried to do, was divide the mgiration up into more sensible amounts. So divide a site into useful sections, minimise cross references and cleanup later. It could be as simple as first importing the users, then importing the content.

Killing the catalog is a good option. As you go along, Plone is merrily indexing and reindexing stuff. Cataloging is a large part of the time. So you could try out switching the catalog off and on again in the transmogrify pipeline.

Watch out for some corners, like that the modification date of all items will be set to today when you are migrating.

Watch out for running out of memory. You may need to restart Zope halfway through so you free up memory, depending on which versions you are using.

A migration can be a good time to get rid of some custom content types, when you have created some new types where in practice the standard types are actually good enough.

There are some dark corners. quintagroup.transmogrifier uses Marshall under the hood. This seemed to be okay, but we ran into some xml problems within Marshall, where the export format was different from the import format. Marshall was last touched in 2006, so it is hard to figure out if there was a reason for this. We have fixed some bugs there.

In netsight.transmogrifier we made some more dynamic pipelines, so look at those.

Takeaway: transmogrifier works. Some glue code needs to be published.

Steve: look at collective.jsonify instead of Marshall. Also: transmogrifying from Archetypes to dexterity works too.

How do you verify that it works? We do manual inspection. You could use web spiders to check things.

Some things do not come over, like the portal_redirection tool and portal_archivist with older revisions.

This is a talk from the Plone conference 2012.

Matt Hamilton and Eric Steele hold the keynote speach on the State of Plone.

Matt Hamilton is the current President of the Plone Foundation. Eric Steele is the Plone 4 release manager.

So, what has happened the last twelve months? Sometimes you think nothing has happened, but when you start writing it down you get really excited because so much is actually going on. This year a Plone Konferenz in Munich, Plone Symposium South America, Plone Open Garden in Sorrento, Plone Symposium East in the US. So several regional conferences in local languages, which is fantastic. In total between five and six hundred attendees. These are grass roots events, not coordinated by the Plone Foundation, which is good. At this Plone Conference, there are people from 27 different countries. We can be proud of our global diversity.

We have had the Plone 4.2 release and 4.3 is now in alpha. A lot is done during sprints. We average one major Plone sprint per month currently. 5597 commits in the past year, 208 contributors. The past month saw 66 people committing to the core of Plone, with 13 of them being new. We will probably break that record this month. We loose some people who move on to new projects, but there are still more people coming in than have left. The Plone core development list is very active.

There are fourteen new members in the Plone Foundation. These are members who have made major contributions to Plone.

World Plone Day had lots of activities worldwide. We can use more events and developers and users in North Africa, Middle East, India.

We had our sixtieth Plone TuneUp online event, shepherded by Carol Ganz from Six Feet Up. Lots of tickets have been closed. In just that last day, 34 tickets have been solved.

We have the Plone Roadmap. The roadmap team created this ongoing document, which is a high level summary of goals of what we want to do with Plone. Some goals may shift. We want to iterate, have more frequent releases. Introduce new technologies early, before they become the standard. We want to be more aggressive about moving under-used and under-maintained features out of core. We want to keep sane upgrades, instead of possibly breaking backwards compatibility in Plone 5.

Plone has moved to github. All of Plone core has moved, making merges, patches, pull requests much easier. Much of the collective has moved too.

There is a new Windows Installer, which is often the first impression a new user gets. Enfold has done a lot of work here. Ross Patterson has been working on getting a much more native installer. The installer uses the same base buildout as other installers. We have been looking at managed cloud deployment options. We have Bitnami, Turnkey, Ploud.

Elizabeth Leddy and Karl Johann Kleist have been cleaning up the Plone Trac, the ticket system, cleaning up tags, reassigned bugs. Fantastic, really helpful. We have made the bug report form simpler. A QA team triages tickets, checking if they are valid and clear.

Timo Stollenwerk leads an effort on automated testing. This is about maintaining code quality. Jenkins is a big part of this. Recently Travis-CI. We are starting to use the Robot Framework, which helps for testing javascript, which is very much needed with the amount of javascript code we have.

This is a talk from the Plone conference 2012.

Jan Jongboom from Cloud9 talks about Javascript.

In 1995 Livescript appeared, developed by Mozilla. A simple scripting language. They called it Javascript, though it was not really like Java at all. It was more like Scheme.

In 1999 Microsoft wanted to build a web version of Outlook 2000. At that point it was not possible to send data from the client to the server without refreshing your whole page. So the XMLHttpRequest was born. This means you can have asynchronous applications.

So Google wanted to build something cool as well. So they launched Gmail in 2004.

Javascript did not get a lot of love in Plone in the beginning. A problem was that there were lots of difference between the Javascript implementation of browsers. The book 'Javascript, the good parts' is eight times smaller than the definitive Guide to Java.

In 2004 (?) JQuery got launched. Maybe the most popular library ever. The power comes from Sizzle, which allows you to easily select DOM elements on your page.

From about 2010 Javascript got more popular than Java, according to some definitions. People started to care. VMs got faster and embeddable. Ecmascript 5 got introduced.

In 2009 the JSConf was the first conference completely about Javascript. NodeJS was introduced at that conference. It allows you to use Javascript on the server. You could think: "Ah, nice, Google's V8 engine as an executable." But it is more powerful than that. LibUV is in there. It abstracts hardware differences away. You would want code to work asynchronously, and operating systems do not normally work that way: you get a file and once you have the data you continue. With LibUV you get events on files, networking, DNS, etcetera: once data starts flowing in, your method gets called so it can continue its work.

Threads can easily get blocked by IO operations. You have a powerful server but due to this, you may still be limited to 300 concurrent connections (if your thread uses ten MB of memory and you have 3GB of memory), or 300 users logged in to the system. With non-blocking threads you can handle tens of thousands of connections.

nodeJS is the Javascript bindings to LibUV. You use standard libraries in Javascript. This Javascript code is then executed in the V8 engine.

Couldn't you use Twisted, the asynchronous Python framework for this? Why would you choose Javascript over Lua or Python or C++? Most Python code is not build to be used asynchronously. In Javascript it is normal.

So we decided to build Cloud9. For most programming languages there are debugging tools and good syntax highlighting. For Javascript: not so much. Cloud9 is an online tool that can do that. You can download it and install it on your server. We give you a free Linux VM with a real terminal. The coolest feature is collaboration. See your colleague type. Debug together.

In our data center we only have Cloud9 code, our IDE server. In another part of the data center we have the remote VMs. We ship it with Python 2.6.6, but you can compile 2.7 or 3.3 on it. You can run MongoDB or Postgres on it, or whatever you want, like Plone. I had problems with it, but you should be fine.

If the standard VM does not have enough power (OpenShift, 128/512 MB Ram, limited resources), you can bring your own VM. I tried the Plone UnifiedInstaller on an Amazon ec2 machine, which I will demonstrate now, if the wireless connection is working good enough. Otherwise a video will have to do. You will see a collaborative session in two browsers where a Plone instance is started and I get a co-worker to assist me on that same machine in his own browser for editing the logo of the Plone instance.

The Cloud9 IDE gives you code completion in the Ace editor, which makes it easier to edit Javascript (like nodeJS) and in a few months Python should be better.

The code is on github so go ahead and fork it.

See http://c9.io/

We use Concorde for the peer to peer messages. Unfortunately closed source.

Could it be used for online sprints, working on some github code of Plone together, assisting someone on the other side of the world? Yes. For open source it is free to use, for closed source you have one free project, and there are paid plans.

There are a couple of issues with the keyboard in the browser, so if you want to use some editor on the command line. You can override some keys in your browser.

Four Digits is organizing the world wide Plone conference 2012 in Arnhem, the Netherlands. Sjoerd welcomes us on behalf of the team.

Michiel van Wessem gives us an introduction to Arnhem. We are at the Musis Sacrum, a music hall. We are very glad this tenth Plone Conference is in our city.

You may know the John Frost bridge, close by. That is known as the 'bridge too far'. A very special place for Arnhem people. If you have time, you should visit the Airborne Museum in Oosterbeek, here in the neighbourhood.

Arnhem is a very green city. In 2009 we were entitled the greenest city of Europe. Our park Sonsbeek has a big part in that. It is very near. Well, everything is very near in Arnhem. There are even some cows near the city center in that park.

We are a city of fashion. Victor and Rolf started their career here. We have the Fashion Biennale, next year the fifth version.

Make sure to visit the city center, with a big variety of shops, restaurants, pubs, for example at the Korenmarkt, with a free WiFi zone.

When you have some time to spend, there are nice things here in Arnhem. On behalf of the city I wish you a very pleasant conference.

Meer info: Nederlandse Plone gebruikersdag 2012.

collective.catalogcleanup

collective.csv2dict

dexterity.membrane

smtp2zope, poi.receivemail

zest.commentcleanup