Python Users Netherlands meeting 21 June 2014
Summary of the meeting of the Dutch Python Users group on 21 June 2014.
Thanks to Reinout van Rees at Nelen & Schuurmans in Utrecht for organizing the meetup.
Tikitu de Jager - async, coroutines, event loops, etc
I work at http://buzzcapture.com
You have an infinite loop with blocking I/O. You do not want that. It could mean you need to work with threads, but you can use coroutines with asyncio. Two infinite loops at the same time, both potentially blocking, both in the same thread. And it all works.
Blocking I/O is like waiting for coffee.
Non-blocking I/O is: place your order, go somewhere else, pick it up when you get called. So: Starbucks.
So: instead of waiting, get out of the way.
This is handled by an event loop. Constantly checking: did anything happen? When anything happens, do I have a callback for it? Compare waiting for a click event in javascript. Be careful not to run into callback hell, otherwise known as callback spaghetti.
A coroutine is a routine that can pause and resume its execution. In python it does yield.
Now you want to be able to yield to the event loop.
You register a subscriber to the event loop. In this subscriber you indicate that you want to yield from somewhere: get some data from somewhere else when an event happens.
The event loop will need to have an API for adding a coroutine to the waiting list and a running list.
Let's compare some aproaches.
Twisted wants to make it easy for you to use networking protocols, with similar ideas. Twisted has been around for a long time, so it uses some different names for similar notions. Porting existing blocking code to Twisted is not always easy.
asyncio has similar high level protocols, but also intends to provide a base layer for other libraries. It uses yield from, which is syntax from Python 3. There is a backport for Python 2 (Tulip), which does not use this syntax.
Remember that you can send() a value to a generator. Also: throw and close. Look those up in the documentation! See also islice for taking a slice of a generator.
When you yield from a generator you hide a loop:
for value in generator:
yield value
With gevent you monkey patch I/O functions:
from gevent import monkey; monkey.patch_all()
gevent uses greenlets, which are full coroutines. Pretty cool. It is a near drop-in in synchronous code. It needs a C extension, so it works only for CPython. Something similar exists for PyPy.
There is also node.js, but I am not going to tell you about that.
You can still use normal callbacks of course.
Audience question: how do tracebacks look with this? Not terrible, but still slightly evil.
See slides.
Reinout van Rees - Python for flood simulation
Reinout works at Nelen & Schuurmans. Demonstration video of a simulation of a flooding in Cape Town. Running in browser, connected with web socket to the server.
Lots of big data. For the Netherlands there is data for the height of every 50 bij 50 centimeter of the country. We use Mercurial to store data.
Calculation core is written in Fortran... We use a Python ctypes wrapper.
Web interface is a Django site with angular.js interface. We have Tornado wrapped around a Django view as a socket interface. The map layers are done with GDAL and Flask. See another talk this evening.
So: Python is great! You can do a lot with it.
Boaz Leskes - python and elasticsearch
Elastic search is a distributed search engine on top of Lucene. Language independent. Json. Can use it for indexing, getting data.
There is a zipfile, you need Java, and then it runs.
Now talk to it with Python:
pip install elasticsearch
There is also a higher level client. See: https://github.com/elasticsearch/elasticsearch-dsl-py
Arjan Verkerk - lightning test
I work here. :-)
You can run a single test file test.py like this:
python -m unittest test
But can I just edit my test file and have the tests run, without me having to alt-tab to go to a different console and run the command again? That is where entr comes in, Event Notify Test Runner:
ls *py | entr -c python -m unittest test
Arjan Verkerk - GDAL raster magic
No, I do not know how to pronounce GDAL properly. ;-)
Tools for georeferenced raster data. python-gdal is the python implementation. Raster data is a raster of images, like those 50 by 50 centimeters of height data of the Netherlands (AHN2).
You need to know your coordinate system. 'Rijksdriehoekstelsel' is good for data in the Netherlands. geotransform can transform coordinates from one system to another.
You can use gdal to store sparse data efficiently. You may have a large area with no data and a small part with actual data.
Demo with AHN2 data.
We use Flask to serve the data.
On the javascript side: leaflets.
Wichert Akkerman - a new simple rest framework
I made REST-toolkit: reinventing the wheel again.
from rest_toolkit import quick_serve, resource
Json responses for all errors. CORS headers. CORS OPTIONS response.
Build on pyramid, so full access to the Pyramid toolset.
Concepts:
- Every url matches a resource.
- Multiple resources can map to the same stored data, but have different permissions and views, say one for anonymous and one for admin.
- Actions are not resources, but handled via a controller, like a remote procedure call, for example to reboot a server. That is not pure REST: not everything is pure data.
SQLAlchemy integration:
from rest_toolkit.ext.sql import SQLResource
A couple of hundred lines of code currently. Coming soon:
- standard views to view/update/delete resources using standard form schema, like json, wtform, Colander, etc.
- Tutorial for AngularJS and other frameworks.
See:
Dennis Kaarsemaker - whelk (makes subprocess easier)
Pretending python is a shell. Sometimes you just want to call a command on the command line and talk to that with python. Can be ugly to write with subprocess, especially when you want to pipe commands together.
Sample code [typos guaranteed, Maurits]:
from whelk import shell, pipe
shell.git(...)
shell['2to3']...
shell.make('test', output_callback=...,
run_callback=..., exit_callback=...
raise_on_error=True)
Easy redirection, per-line logging, encodings.
git = shell.git
git.checkout('next')
if not git.diff('--quiet', 'master', raise_on_error=True):
...
pip install whelk
Marcel van den Elst - complete RESTful stack from pyramid and mongoengine to backbone
Example: ZooStack for managing RESTful animals front to back.
At http://progressiveplanning.com we use Pyramid, we came from Django and TastyPIE. We transformed it into something that MongoDB could use.
A RESTful stack from front to back.
- Flexible, long-lived MVC client application framework
- well defined, related and privilege-checked documents
- secure RESTful API producer and consumer
MongoEngine is nearly dead, do not use it. So we are going to have to fix some things ourselves. Document definitions.
TastyMongo is a RESTful API for MongoEngine, a la Tastypie.
Backbone-Relational. Useful when an action triggers changes in multiple models on the client and the server. You can try to do it manually if you want, but Backbone-Relational takes care of it for you. See http://backbonerelational.org