Òscar Vilaplana (Paylogic) talks about ØMQ (Zero MQ), at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.

0MQ is sockets how it should be. Bindings are available in many, many languages, including Python. Of course messaging looks simple: you send a message and the guy on the other end receives it! Well, it still requires work. But 0MQ is indeed simple. (In python: import zmq). Messages are sent in the background. You have a queue. If the receiver is not there, the message just stays in the queue longer.

Messages are strings and they have a length. With multiparts they can have a sender. You can send through TCP or UDP. You can publish and subscribe. You send messages down a pipeline. As infrastructure you can choose a queue, a forwarder or a streamer.

You can poll from a 0MQ socket or a regular TCP socket or stdin.

Code from this talk: http://oscarvilaplana.cat/zmqtalk.tar.gz

Pieter Noordhuis: Redis in Practice, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.

Redis is a key-value store. It can be compared to memcached. But it natively supports strings, lists, sets, sorted sets and hashes. Everything is stored in memory, so that puts a limit on what you can put in it but also makes it very fast. You can also persist it though, unlike memcached. Supports replication, so you can have one master that you write to and say fifty slaves just for reading.

Any blob will do: ascii, utf-8, png. Example:

redis> set str "hello world"
OK
redis> get str
"hello world"

It runs in a single thread: no race conditions or locks; every operation is atomic. This greatly simplifies replication.

Invalidate immediately:

redis> del page:/home
(integer) 1

You can have rate limiting:

INCR limit
EXPIRE limit 60 # iff INCR == 1

Lists are a natural fit: RPUSH a new job (push it at the right of the list) and LPOP a job from the left of the list. With PubSub you can set up some notifications when jobs are done.

Sets: unordered sets of unique values. Ordered Sets of unique values: ZADD an item to increase its score, ZREM to decrease it. You can use this easily to show currently logged in users, top users for some measurement, etc.

We can do 100,000 gets and sets per second on commodity hardware. With tweaking and better hardware we have heard of 1.5 million per second.

You can have durability through snapshotting: save every N seconds or every N changes.

Getting started: http://redis.io/download, no dependencies, make it and you are done.

Henk Doornbos (Paylogic) talks about making large, untested code bases testable, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.

Computer scientist, software engineer, architect, consultant, head architecture department. The Paylogic system is front and back office for payment. Mostly Python (84406 lines of code) and javascript.

There are many abstraction levels: components, classes, features, the code level. Process when doing a support case: staging, testing, bug reports. We use FogBugz as bug tracker. Example: downloading a 'scanware' file was broken. Problem solved now, but can we check automatically that it has indeed been fixed and is not broken later again. Maybe we can execute the bug report as a test? This can be done when you are rigid enough in specifying the bug report ('Given When Then' style: starting state, steps to reproduce, expected behavior). You can write a small language that specifies what should be done when you encounter specific text in a bug report, translating that to Selenium tests (visit this page, click there, check that text is on the resulting page). We use the robot framework.

You could do similar things with record and playback. But often the bug report is a missing requirement. We only want to add features that deliver value. Behavior driven development style: to have some fun (goal, value), as a customer (who, actor), I want to buy a ticket (reason). You can write that as:

• Given I am buying tickets
• When I choose the amount
• Then I see the total cost

So in one smooth, agile process you create these products: requirements document, acceptence tests, working code that delivers value, bug reports.

Gideon de Kok talks about Mobile Architectures, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.

Yes, you should treat mobile clients differently. Mobile applications should still work reasonably even when not connected, if possible. You want to get things done quickly. More push and pull. Best practices are still mostly the same. Difference: your server should work harder than the client: no big calculations in javascript. Don't let the client pull but let the server push; this saves battery life.

Improve connectivity: do lots of caching. Store them locally in the browser/app. Refresh your data periodically, also on user request. Properly expire and delete data. Combine requests so you have less problems with latency and connectivity. If connection fails, just wait. Queue requests and send them when connectivity is there again, instead of checking every few seconds. Restrict connectivity: if you don't use it, don't send it. Don't preload things the client may possibly need.

Make your payload small, one simple trick is of course using gzip. Adapt your payload: send smaller images or send only alternative text. No more direct database access: create an API to talk to the database, including logic for failing connectivity. Never trust API input. Client side validation is good for connectivity and performance, but you definitely need to check on the server too.

Security: encrypt your data streams and store secret stuff well. And do not save too secret stuff on the client: you don't want thieves stealing secrets from your phone by reverse engineering.

A mobile-specific API probably has more push and less pull calls.

Don't overengineer your SSL connections. It takes too long to decrypt it on a mobile device if you send too much encrypted data. Safety versus speed.

Native applications are generally speedier and better in the pushing and pulling, relying less on good connectivity than a mobile web site. When you build native apps as e.g. a bank, you suddenly become a software creator, instead of just needing to keep a website running. A web app based on html5 and css3 it will mostly work on all devices; with a mobile specific app, you have lots more devices that you would need to check for compatibility. There are frameworks for this, but the resulting apps will be less effective than an app created specifically for one device.

Luit van Drongelen talks about Lightweight Python deployment servers, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.

I code python for fun (and hopefully eventually for profit). What is wrong with Apache? Well, nginx is much faster. You can double the responses and have much less memory usage. Apache has higher I/O load, e.g. for loading .htaccess file. nginx can natively connect to uWSGI: fast, light weight version of modwsgi. The protocol is uwsgi (so all lowercase). This is run separately from the web server. Tested on many operating system; not Windows, sadly.

So why use uWSGI instead of e.g. modwsgi. It's fast with a lower memory foot print. Can handle multiple interpreteer versions in multiple virtualenvs. Supports old and new WSGI standard. It can kill misbehaving worker threads; helps if you are coding those wrongly yourself. It can also handle long-running tasks. Configuration can be in ini files, json, environment variables, command line options, etcetera. Built-in message-passing system. Embedded (evented/async) HTTP server. It even has a clustering feature (beta at the moment). [Live demo of that feature, which worked yesterday.]

Some other WSGI servers show great performance too. For me uWSGI performs great and has good features.

Comment from room: look at this WSGI performance comparison.

See the WSGI slides.