When to use jsonmigrator

• Good for migrating from old versions of Plone where you cannot install transmogrifier for exporting the contents.
• Even from old Zope/CMF sites.
• Particularly when you have no buildout.
• When changing from archetypes to dexterity.
• To clean cruft from an error-prone old site.
• When scared to upgrade because you don't know what might bite you later. You don't know what you don't know...

Note: this is not the whole story: you still need to update your Plone version, add-ons, custom code, theme.

When NOT to use transmogrifier

• From Plone 4.x.
• Probably not from 3.3.6 where you already have buildout: just use the standard Plone upgrade path.

Technology stack

• plone.app.transmogrifier + collective.transmogrifier as base
• collective.jsonify
• collective.jsonmigrator
• extra pipelines:
  • transmogrify.dexterity
  • quintagroup.transmogrifier (I had some issues with it, but it has useful stuff, so you may need it.)

Setting up jsonmigrator

• Duplicate your old Plone in your staging environment. Or use your public live site, but that is not recommended: it may have security issues, or you may run into corner cases where your site breaks. So do it on a copy please.
• Install collective.jsonify there. If you don't have buildout, you may need to do it like this:
  • Download the egg to your Plone directory
  • Unzip it
  • Add it to your Python with setup.py install, which will also pull in simplejson. Use a virtualenv if possible.
• Add some external methods in an Extensions directory, where you import the main jsonify functions. Add and use the methods in the ZMI of your old Plone Site. For example the get_item function, which you can then add to the end of a url to get the json version of content item. For a file you get the data included in base64 encoding.
• Add collective.jsonmigrator to the eggs of your new Plone Site.
• Go to the @@jsonmigrator view. Select a pipeline, update any settings, and run it.

In our case we had setup a really long pipeline to migrate from a Zope 2.6 site to Plone 4.

You can define your own pipeline sections, usually based on collective.transmogrifier.sections.manipulator to tweak data in the dictionary of the current item or .inserter to add data. Start with the collective.jsonmigrator.remotesource section for getting the data.

You may have to get your hands dirty when things don't exactly work the first time. Adding some print statements can already help to get the problem clear.

Caveats

• It does not export/import users, though see collective.blueprint.usersandgroups.
• Doesn't do portlets.
• collective.jsonify is not security safe to use at a live site, as mentioned.

Credits

• Transmogrifier: Martijn Pieters, Jarn.
• Jsonmigrator: Rok Garbas
• RCS: letting me loose on this
• Calvin and Hobbes: Bill Watterson, building the first migrator.

Warning from Steve: I have run into problems with character sets. Paul: there are encoding and decoding blueprints that help, though it takes some getting.

Maarten: You are getting json data from an old site that is still running. Can't you save the json data on disk and use that? Erico: yes, you can, that is what I usually do. Maarten: I am using a blueprint for that, also because sometimes the export gave an error and sometimes the import.

Watch the video of this talk.

Tikitu de Jager - async, coroutines, event loops, etc

I work at http://buzzcapture.com

You have an infinite loop with blocking I/O. You do not want that. It could mean you need to work with threads, but you can use coroutines with asyncio. Two infinite loops at the same time, both potentially blocking, both in the same thread. And it all works.

Blocking I/O is like waiting for coffee.

Non-blocking I/O is: place your order, go somewhere else, pick it up when you get called. So: Starbucks.

So: instead of waiting, get out of the way.

This is handled by an event loop. Constantly checking: did anything happen? When anything happens, do I have a callback for it? Compare waiting for a click event in javascript. Be careful not to run into callback hell, otherwise known as callback spaghetti.

A coroutine is a routine that can pause and resume its execution. In python it does yield.

Now you want to be able to yield to the event loop.

You register a subscriber to the event loop. In this subscriber you indicate that you want to yield from somewhere: get some data from somewhere else when an event happens.

The event loop will need to have an API for adding a coroutine to the waiting list and a running list.

Let's compare some aproaches.

Twisted wants to make it easy for you to use networking protocols, with similar ideas. Twisted has been around for a long time, so it uses some different names for similar notions. Porting existing blocking code to Twisted is not always easy.

asyncio has similar high level protocols, but also intends to provide a base layer for other libraries. It uses yield from, which is syntax from Python 3. There is a backport for Python 2 (Tulip), which does not use this syntax.

Remember that you can send() a value to a generator. Also: throw and close. Look those up in the documentation! See also islice for taking a slice of a generator.

When you yield from a generator you hide a loop:

for value in generator:
    yield value

With gevent you monkey patch I/O functions:

from gevent import monkey; monkey.patch_all()

gevent uses greenlets, which are full coroutines. Pretty cool. It is a near drop-in in synchronous code. It needs a C extension, so it works only for CPython. Something similar exists for PyPy.

There is also node.js, but I am not going to tell you about that.

You can still use normal callbacks of course.

Audience question: how do tracebacks look with this? Not terrible, but still slightly evil.

See slides.

See Maurits (1 and 2) and Reinout for PyGrunn summaries.

Reinout van Rees - Python for flood simulation

Reinout works at Nelen & Schuurmans. Demonstration video of a simulation of a flooding in Cape Town. Running in browser, connected with web socket to the server.

Lots of big data. For the Netherlands there is data for the height of every 50 bij 50 centimeter of the country. We use Mercurial to store data.

Calculation core is written in Fortran... We use a Python ctypes wrapper.

Web interface is a Django site with angular.js interface. We have Tornado wrapped around a Django view as a socket interface. The map layers are done with GDAL and Flask. See another talk this evening.

So: Python is great! You can do a lot with it.

Boaz Leskes - python and elasticsearch

Elastic search is a distributed search engine on top of Lucene. Language independent. Json. Can use it for indexing, getting data.

There is a zipfile, you need Java, and then it runs.

Now talk to it with Python:

pip install elasticsearch

There is also a higher level client. See: https://github.com/elasticsearch/elasticsearch-dsl-py

Arjan Verkerk - lightning test

I work here. :-)

You can run a single test file test.py like this:

python -m unittest test

But can I just edit my test file and have the tests run, without me having to alt-tab to go to a different console and run the command again? That is where entr comes in, Event Notify Test Runner:

ls *py | entr -c python -m unittest test

See http://entrproject.org

Arjan Verkerk - GDAL raster magic

No, I do not know how to pronounce GDAL properly. ;-)

Tools for georeferenced raster data. python-gdal is the python implementation. Raster data is a raster of images, like those 50 by 50 centimeters of height data of the Netherlands (AHN2).

You need to know your coordinate system. 'Rijksdriehoekstelsel' is good for data in the Netherlands. geotransform can transform coordinates from one system to another.

You can use gdal to store sparse data efficiently. You may have a large area with no data and a small part with actual data.

Demo with AHN2 data.

We use Flask to serve the data.

On the javascript side: leaflets.

Wichert Akkerman - a new simple rest framework

I made REST-toolkit: reinventing the wheel again.

from rest_toolkit import quick_serve, resource

Json responses for all errors. CORS headers. CORS OPTIONS response.

Build on pyramid, so full access to the Pyramid toolset.

Concepts:

• Every url matches a resource.
• Multiple resources can map to the same stored data, but have different permissions and views, say one for anonymous and one for admin.
• Actions are not resources, but handled via a controller, like a remote procedure call, for example to reboot a server. That is not pure REST: not everything is pure data.

SQLAlchemy integration:

from rest_toolkit.ext.sql import SQLResource

A couple of hundred lines of code currently. Coming soon:

• standard views to view/update/delete resources using standard form schema, like json, wtform, Colander, etc.
• Tutorial for AngularJS and other frameworks.

See:

https://github.com/wichert/rest_toolkit

http://rest-toolkit.rtfd.org

Dennis Kaarsemaker - whelk (makes subprocess easier)

Pretending python is a shell. Sometimes you just want to call a command on the command line and talk to that with python. Can be ugly to write with subprocess, especially when you want to pipe commands together.

Sample code [typos guaranteed, Maurits]:

from whelk import shell, pipe
shell.git(...)
shell['2to3']...
shell.make('test', output_callback=...,
           run_callback=..., exit_callback=...
           raise_on_error=True)

Easy redirection, per-line logging, encodings.

git = shell.git
git.checkout('next')
if not git.diff('--quiet', 'master', raise_on_error=True):
    ...

pip install whelk

See https://github.com/seveas/whelk

Marcel van den Elst - complete RESTful stack from pyramid and mongoengine to backbone

Example: ZooStack for managing RESTful animals front to back.

At http://progressiveplanning.com we use Pyramid, we came from Django and TastyPIE. We transformed it into something that MongoDB could use.

A RESTful stack from front to back.

• Flexible, long-lived MVC client application framework
• well defined, related and privilege-checked documents
• secure RESTful API producer and consumer

MongoEngine is nearly dead, do not use it. So we are going to have to fix some things ourselves. Document definitions.

TastyMongo is a RESTful API for MongoEngine, a la Tastypie.

Backbone-Relational. Useful when an action triggers changes in multiple models on the client and the server. You can try to do it manually if you want, but Backbone-Relational takes care of it for you. See http://backbonerelational.org

Part 1: Single Sign-on (SSO) using a Central Authentication Service (CAS).

Without SSO you need to login manually to lots more websites or apps, or put everything in one website/app. Or you can create a custom API to share authentication information.

With SSO a user logs in once and gains access to all systems without begin prompted to log in again.

(JaSig) CAS is an enterprise SSO solution. Open source, well documented. Web browser talks to application server, app server talks to CAS, web browser authenticates with CAS once, CAS gives the app a ticket showing that the web browser is authenticated.

See http://jasig.org/cas

Pro: CAS is modular, highly pluggable, so it is fit for lots of frameworks.

Contra: SSO availability and security become critical. You should subscribe to the CAS mailing list to keep on top of issues. Setup notifications for when your CAS server runs into problems.

Our use case:

• User base in an Active Directory server
• CAS server
• Two apps/websites:
  • Dashboard app in Django
  • Document management system in Plone
  • More to come

CAS alternatives: JOSSO, OpenAM, Pubcookie, CoSign.

Part 2: Two-step verification

Or: two-factor authentication. Can be based on:

• knowledge factor, something only you know
• possession factor, something only you possess
• inheritance factor, something only you have, like fingerprints

Common solutions:

• SMS authentication. You need to pay for each SMS, so it can become expensive. Phone might not be connected.
• Google Authenticator (for mobile apps). Very easy to integrate in your app. Requires an extra app to be installed on your device.
• Hardware token generators. Easy to use. But extra device to carry with you.

Software we made for Plone:

• collective.googleauthenticator
• collective.smsauthenticator

Similar software is available for Django.

Alternatives:

• risk-based authentication, based on behavioral biometrics, keystroke dynamics, etc
• strong authentication
• reliance authentication