Sylvain Viollon: Tornado and IO in Python 3

See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.

I work at MindDistrict. Today I will talk about a web service I built with Tornado. It needs to talk to lots of things, without doing much computation itself. It uses Python 3, because it was about time to use it.

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado is IO loop based. Does not work with a WSGI server. It is easy to implement (micro) web services.

What is an IO loop? Old programming technique, from when threads did not exist or were expensive. It does not use threads, only one flow to execute the code. The loop calls you, you do things, when you are done you relinquish control to the loop. Not efficient for computation intensive tasks.

You can use callbacks, but they are ugly. With generators for the IO loop you can replace them. With @tornado.gen.coroutine you decorate a function and let it yield results.

It works with 'future' objects: a result that still needs to be computed. Comparable with promises in Javascript. You can create them yourself:

def some_function():
    future = tornado.concurrent.Future()
    # do stuff
    future.set_result(42)
    return future

Tornado provides an improved version of subprocess.pOpen. Create a future, call the subprocess, when it is done set the return value or an exception.

You cannot use normal threads or file locks. But you can implement a lock mechanism with futures.

Future objects are reusable for caching. 'memoize' type decorators work.

concurrent.futures is part of the standard library in Python 3. With it you can run your heavy computations in a separate process. It works with tornado: create a coroutine and use the Python 3 futures in there. There is a backport for Python 2.7.

Then Python 3. The syntax for generators is much better. Enough support in main packages. It works nice for us in production.