Current state

js registry, css registry, tools. Register your files in cssregistry.xml and jsregistry.xml. There are limitations. We are missing out:

• js and css have evolved. No one wanted to touch javascript a few years ago and now it is quite fun.
• dependency management was added in the javascript world
• now there is precompiling with less/sass
• versioning
• Mockup to do this in Plone.

Other tools

• RequireJS: you include require.js, add all your dependencies in javascript, asynchronous loading of files and modules.
• grunt: task runner, run tests, concat, minimize, jshint, etc. Kind of like a Makefile, but more suited for Javascript.
• npm: nodeJS server based package management tool, dependencies, versions.
• Bower: same thing on client side.

Less/sass: precompilers for css, variables, nesting, macros.

Design of the solution

• Diazo on top of static html
• bundle, resource, registry

Bundles are the main resources, loaded in dependency order. They can have js and less files that are compiled. You can enable/disable a bundle. There is a special legacy bundle with the old stuff as you are used to in current Plone 4.

Resource: defines the js, any needed export/init code, css, url. All defined in the Plone configuration registry.

Define Less variables, in the config registry record plone.lessvariables. For example define your own favorite plone-link-color.

Pattern options, for example for the pickadate widget.

Development versus production

• Compile on browser on each reload using RequireJS/Less.
• Develop static html with compiled bundles for testing
• Grunt using browser config.js/mixins.less.

Diazo

You can now say which bundles are enabled.

Views

• config.js has requireJS config.
• less-variables.js: has the standard less variables.
• plone.less: import files, patterns
• plone.js: requires the files.

This all is fully customizable through the web.

Legacy Plone: cssregistry.xml and jsregistry.xml still work. You probably want to convert it though. There are some incompatibilities: there is no ordering in the new stuff, only dependencies. So you may need to tweak a bit.

generate_gruntfile.py extracts all the information from the resource registries and creates a Gruntconfig.js file so you can test it on a static site.

A new Plone 5 site uses production settings for the theme. For development you obviously want to put it in development mode in the Site Setup. You can edit files through the web, have it compile and see the changes.

Migration

• Define resources as pattern and less file.
• Always generate production bundle.
• Move jsregistry.xml and cssregistry.xml to registry.xml.
• On/live jquery 1.11 modifications.
• Never loose hope.

See a plone.app.mosaic branch for the changes I did for migrating that package. It took me less than one hour to migrate.

More information: http://www.nathanvangheem.com/news/plone-5-resource-registries

We are going to sprint on this.

Watch the video of this talk.

How to transfer a prototype into a Diazo theme for Plone

• Prototype based on the default Plone markup as far as possible.
• Know the limits of prototyping and know when to stop and use the real site.
• Use Diazo to move blocks around, do easy modifications to the default markup. Do not forget to make space for the toolbar just after the tag.
• Only if strictly needed, modify existing templates to match the prototype ones, using z3c.jbot.
• Use iterative improvement. Demo it to the customer frequently.
• Add custom styling for standard user generated markup in TinyMCE.

main_template is now a browser view, instead of a skins resource: Products.CMFPlone/browser/main_template.py.

Resource registries have been completely updated for modern front-end developing. Plone bundle, Barceloneta theme bundle, Legacy bundle. In your Diazo theme in manifest.cfg you can enable or disable bundles.

Development mode: compilation on browser of less/js resources. Compilation on console via grunt watch.

Barceloneta theme

• Pretty well ordered files: plonetheme > barceloneta > static > less.
• Deductible LESS structure and file naming.
• You can reuse the Barceloneta LESS files in your own theme by importing it with the sitePath variable.
• We did not use much Diazo: the rules file is very simple.

Toolbar is not completely isolated, we ran into problems using an iframe for this, it is still the plan for the future, it is now about halfway.

Maybe we can add a checkbox for developing in LESS and not Javascript and the other way around, so your browser does not need to compile everything all the time.

Twitter: @albertcasado and @sneridagh.

Watch the video and slides of this talk.

When to use jsonmigrator

• Good for migrating from old versions of Plone where you cannot install transmogrifier for exporting the contents.
• Even from old Zope/CMF sites.
• Particularly when you have no buildout.
• When changing from archetypes to dexterity.
• To clean cruft from an error-prone old site.
• When scared to upgrade because you don't know what might bite you later. You don't know what you don't know...

Note: this is not the whole story: you still need to update your Plone version, add-ons, custom code, theme.

When NOT to use transmogrifier

• From Plone 4.x.
• Probably not from 3.3.6 where you already have buildout: just use the standard Plone upgrade path.

Technology stack

• plone.app.transmogrifier + collective.transmogrifier as base
• collective.jsonify
• collective.jsonmigrator
• extra pipelines:
  • transmogrify.dexterity
  • quintagroup.transmogrifier (I had some issues with it, but it has useful stuff, so you may need it.)

Setting up jsonmigrator

• Duplicate your old Plone in your staging environment. Or use your public live site, but that is not recommended: it may have security issues, or you may run into corner cases where your site breaks. So do it on a copy please.
• Install collective.jsonify there. If you don't have buildout, you may need to do it like this:
  • Download the egg to your Plone directory
  • Unzip it
  • Add it to your Python with setup.py install, which will also pull in simplejson. Use a virtualenv if possible.
• Add some external methods in an Extensions directory, where you import the main jsonify functions. Add and use the methods in the ZMI of your old Plone Site. For example the get_item function, which you can then add to the end of a url to get the json version of content item. For a file you get the data included in base64 encoding.
• Add collective.jsonmigrator to the eggs of your new Plone Site.
• Go to the @@jsonmigrator view. Select a pipeline, update any settings, and run it.

In our case we had setup a really long pipeline to migrate from a Zope 2.6 site to Plone 4.

You can define your own pipeline sections, usually based on collective.transmogrifier.sections.manipulator to tweak data in the dictionary of the current item or .inserter to add data. Start with the collective.jsonmigrator.remotesource section for getting the data.

You may have to get your hands dirty when things don't exactly work the first time. Adding some print statements can already help to get the problem clear.

Caveats

• It does not export/import users, though see collective.blueprint.usersandgroups.
• Doesn't do portlets.
• collective.jsonify is not security safe to use at a live site, as mentioned.

Credits

• Transmogrifier: Martijn Pieters, Jarn.
• Jsonmigrator: Rok Garbas
• RCS: letting me loose on this
• Calvin and Hobbes: Bill Watterson, building the first migrator.

Warning from Steve: I have run into problems with character sets. Paul: there are encoding and decoding blueprints that help, though it takes some getting.

Maarten: You are getting json data from an old site that is still running. Can't you save the json data on disk and use that? Erico: yes, you can, that is what I usually do. Maarten: I am using a blueprint for that, also because sometimes the export gave an error and sometimes the import.

Watch the video of this talk.

Tikitu de Jager - async, coroutines, event loops, etc

I work at http://buzzcapture.com

You have an infinite loop with blocking I/O. You do not want that. It could mean you need to work with threads, but you can use coroutines with asyncio. Two infinite loops at the same time, both potentially blocking, both in the same thread. And it all works.

Blocking I/O is like waiting for coffee.

Non-blocking I/O is: place your order, go somewhere else, pick it up when you get called. So: Starbucks.

So: instead of waiting, get out of the way.

This is handled by an event loop. Constantly checking: did anything happen? When anything happens, do I have a callback for it? Compare waiting for a click event in javascript. Be careful not to run into callback hell, otherwise known as callback spaghetti.

A coroutine is a routine that can pause and resume its execution. In python it does yield.

Now you want to be able to yield to the event loop.

You register a subscriber to the event loop. In this subscriber you indicate that you want to yield from somewhere: get some data from somewhere else when an event happens.

The event loop will need to have an API for adding a coroutine to the waiting list and a running list.

Let's compare some aproaches.

Twisted wants to make it easy for you to use networking protocols, with similar ideas. Twisted has been around for a long time, so it uses some different names for similar notions. Porting existing blocking code to Twisted is not always easy.

asyncio has similar high level protocols, but also intends to provide a base layer for other libraries. It uses yield from, which is syntax from Python 3. There is a backport for Python 2 (Tulip), which does not use this syntax.

Remember that you can send() a value to a generator. Also: throw and close. Look those up in the documentation! See also islice for taking a slice of a generator.

When you yield from a generator you hide a loop:

for value in generator:
    yield value

With gevent you monkey patch I/O functions:

from gevent import monkey; monkey.patch_all()

gevent uses greenlets, which are full coroutines. Pretty cool. It is a near drop-in in synchronous code. It needs a C extension, so it works only for CPython. Something similar exists for PyPy.

There is also node.js, but I am not going to tell you about that.

You can still use normal callbacks of course.

Audience question: how do tracebacks look with this? Not terrible, but still slightly evil.

See slides.

See Maurits (1 and 2) and Reinout for PyGrunn summaries.

Reinout van Rees - Python for flood simulation

Reinout works at Nelen & Schuurmans. Demonstration video of a simulation of a flooding in Cape Town. Running in browser, connected with web socket to the server.

Lots of big data. For the Netherlands there is data for the height of every 50 bij 50 centimeter of the country. We use Mercurial to store data.

Calculation core is written in Fortran... We use a Python ctypes wrapper.

Web interface is a Django site with angular.js interface. We have Tornado wrapped around a Django view as a socket interface. The map layers are done with GDAL and Flask. See another talk this evening.

So: Python is great! You can do a lot with it.

Boaz Leskes - python and elasticsearch

Elastic search is a distributed search engine on top of Lucene. Language independent. Json. Can use it for indexing, getting data.

There is a zipfile, you need Java, and then it runs.

Now talk to it with Python:

pip install elasticsearch

There is also a higher level client. See: https://github.com/elasticsearch/elasticsearch-dsl-py

Arjan Verkerk - lightning test

I work here. :-)

You can run a single test file test.py like this:

python -m unittest test

But can I just edit my test file and have the tests run, without me having to alt-tab to go to a different console and run the command again? That is where entr comes in, Event Notify Test Runner:

ls *py | entr -c python -m unittest test

See http://entrproject.org

Arjan Verkerk - GDAL raster magic

No, I do not know how to pronounce GDAL properly. ;-)

Tools for georeferenced raster data. python-gdal is the python implementation. Raster data is a raster of images, like those 50 by 50 centimeters of height data of the Netherlands (AHN2).

You need to know your coordinate system. 'Rijksdriehoekstelsel' is good for data in the Netherlands. geotransform can transform coordinates from one system to another.

You can use gdal to store sparse data efficiently. You may have a large area with no data and a small part with actual data.

Demo with AHN2 data.

We use Flask to serve the data.

On the javascript side: leaflets.

Wichert Akkerman - a new simple rest framework

I made REST-toolkit: reinventing the wheel again.

from rest_toolkit import quick_serve, resource

Json responses for all errors. CORS headers. CORS OPTIONS response.

Build on pyramid, so full access to the Pyramid toolset.

Concepts:

• Every url matches a resource.
• Multiple resources can map to the same stored data, but have different permissions and views, say one for anonymous and one for admin.
• Actions are not resources, but handled via a controller, like a remote procedure call, for example to reboot a server. That is not pure REST: not everything is pure data.

SQLAlchemy integration:

from rest_toolkit.ext.sql import SQLResource

A couple of hundred lines of code currently. Coming soon:

• standard views to view/update/delete resources using standard form schema, like json, wtform, Colander, etc.
• Tutorial for AngularJS and other frameworks.

See:

https://github.com/wichert/rest_toolkit

http://rest-toolkit.rtfd.org

Dennis Kaarsemaker - whelk (makes subprocess easier)

Pretending python is a shell. Sometimes you just want to call a command on the command line and talk to that with python. Can be ugly to write with subprocess, especially when you want to pipe commands together.

Sample code [typos guaranteed, Maurits]:

from whelk import shell, pipe
shell.git(...)
shell['2to3']...
shell.make('test', output_callback=...,
           run_callback=..., exit_callback=...
           raise_on_error=True)

Easy redirection, per-line logging, encodings.

git = shell.git
git.checkout('next')
if not git.diff('--quiet', 'master', raise_on_error=True):
    ...

pip install whelk

See https://github.com/seveas/whelk

Marcel van den Elst - complete RESTful stack from pyramid and mongoengine to backbone

Example: ZooStack for managing RESTful animals front to back.

At http://progressiveplanning.com we use Pyramid, we came from Django and TastyPIE. We transformed it into something that MongoDB could use.

A RESTful stack from front to back.

• Flexible, long-lived MVC client application framework
• well defined, related and privilege-checked documents
• secure RESTful API producer and consumer

MongoEngine is nearly dead, do not use it. So we are going to have to fix some things ourselves. Document definitions.

TastyMongo is a RESTful API for MongoEngine, a la Tastypie.

Backbone-Relational. Useful when an action triggers changes in multiple models on the client and the server. You can try to do it manually if you want, but Backbone-Relational takes care of it for you. See http://backbonerelational.org