Reinout van Rees - Report on testing

Testing is good, but not everyone does it. How about making a report of who is actually writing tests and which project has commits with tests? With githubinfo you get a report of your projects on github sorted by the amount of commits that have tests in them. I send an email out to my colleagues with that info every week. The email also contains a list of committers sorted by the amount of commits with tests. This should help in getting people more aware of testing. You can configure it with a json file.

https://github.com/nens/githubinfo http://pypi.python.org/pypi/githubinfo

Ronald Evers - Dev'ing at Fox, how do they do it

I will show you a little bit about how we do development and operations here at Fox-IT. I am working on DetACT, fraud detection software for online banking. We try to detect fraud in the data stream. Low-level in C++, the rest in Python. We have sold DetACT to several banks.

We switched from subversion to git. We use Gerrit. This is a code review system. Before you are allowed to push to the master branch, you need to get two upvotes for your commits, which is done on the Gerrit server. You can compare it a bit with a pull request on github.

We use Jenkins for automated tests. This also works nicely in combination with Gerrit.

We have to respond quickly to new security attacks, so we deploy as often as possible. We use irc, Fabric, Sentry, Munin, Nagios, request tracker. Sentry is an exception tracker: what is going wrong on which server? We have built a simple irc bot for Gerrit that tells us on an irc channel what is going on with the code.

For writing an irc bot: https://github.com/ronaldevers/botlerplate

Alternatively: https://github.com/github/hubot

With Fabric we execute actions on several servers. It is like Make, but also works on remote servers. Selecting which hosts you want to deploy on is actually the hardest part for us. So we have built something for that. We change the globals() dynamically to add extra targets. It works.

JSON, HTTP and the requests library are your friends. Connect everything. Automate everything. Monitor everything.

Yes, we are hiring.

Jeroen Vloothuis - Get more attention for your project

Do not let your open source work lead to nothing. Or: how to let your project get the deserved attention.

A lot of people do open source projects, but they make a few mistakes in communication and the project does not succeed. So a few simple marketing tips to help you maximize your user audience.

People care about themselves. In documentation talk to the user: "This software lets you write efficient code." Get your users engaged. Emphasize what it does for them.

"Your project solves my pain, right?" That is what the user wants to know. Why would you write software when not to solve a real problem? "Django makes it easier to build better web apps more quickly with less code." Sounds good. Compare that with: "Flask is based on Werkzeug and Jinja2 and has a BSD license." Yeah? So? What can I actually do with it?

So: why did you start your project and what problems does it solve?

Talk about benefits, not features. "My project has a FrobNob 2000." Right. "My project saves you time." That could be useful. "Rely on MySQL to save time and money." Good. "We are blah compliant, have full support for blah." Excuse me? If you are ACID compliant, say something like "we will not corrupt your data."

So: focus on benefits. Features just support your claim.

A picture is worth a thousand words. In the case of Python the picture can be a few lines of source code, showing how easy it is to use your package. Show, don't tell.

So focus on:

• your users
• their pains
• benefits first
• pictures.

Question: Won't developers like the facts more? They do want to know if a database is ACID compliant.

Answer: Fine, but still start with telling that in simple terms.

Jasper Spaans - A talk about nothing

This is a lightning talk about nothing. I counted the zeroes and ones in the DetACT project. The zeroes clearly win, with almost sixty percent.

See https://github.com/jap/bittally

PEP-336: make None a callable object. That would be handy, as that would mean you can do None.endswith('wawawa') and it won't fail so you need less code. Guido denied this pep. I wrote this code if you want to use it anyway.

See https://github.com/jap/nihilo

Onno Broekmans - Programming scientists

PhD student at University of Amsterdam. I am wondering who you guys (and girls!) are. Lots of web development here, I think. I am working as a scientist. Scientists have invented the atom bomb, particle accelerator, and we do it with spaghetti code. Most scientists have no background in programming. We teach it to ourselves. We mess around a bit. What is version control? Bug tracking? Unit testing? Never heard of it.

But we are supposed to cure cancer and figure out this climate thing. Scientists should be educated in programming and the activities around it. This is not easy to do, but we are working on it. In Canada there are a few guys that do Software Carpentry. They are coming to the Netherlands. If you are a scientist, please join them. If you have ideas about how to teach people about Python, I would be interested to hear about that. Are you interested in being involved, talk to me.

Not many code from scientists makes it to the community. We should, but we don't. It is the elephant in the room.

Jan-Jaap Driessen - Fanstatic 1.0a

fanstatic takes the pain out of static resources. Your cool web application does a lot of difficult computations and a lot of dumb stuff: serving Javascript and css files.

Fanstatic helps you there, with lots of js.* packages on http://pypi.python.org. You can do your version requirements for javascript files in the setup.py of your python package.

You can bundle resources into one. Since last night it has compilation support, so coffeescript, sass, less, clojure, etcetera. Fanstatic saves you time. So if you write sass you no longer need to compile it yourself, as fanstatic does it for you.

You no longer need to think about getting all your javascript dependencies in the html page. You let Python and fanstatic handle it.

Did you see me compile anything during this demo, and bundle those compiled resources, and minify those bundled resources? No! I save so much time and have so much less pain! Fanstatic saves the day.

You may not want to install all the compilers and minifiers on your server and fanstatic can help you there too.

It works with WSGI (fanstatic is the middleware), Zope, flash, Django.

Bundling is a hard topic. It depends on your stack and your setup.

Question: when would you not use fanstatic?

Answer: when I have a lot of time.

We support injection in the html head or at the bottom if your resource can handle that.

There is a branch for loading resources from a content delivery network.

http://www.fanstatic.org/

Orne Brocaar - Job-Runner

Job-Runner is a product we built inside Spil. I am a freelance Python/Django web application developer. I am currently working for Spil Games, a gaming company in Hilversum, about 70 people working there, about 50 gaming portals worldwide.

I am working within the data warehouse team. We gather data from internal and external sources, like analytics data, game related data. We convert this into a Data Vault and Data Mart model, where we can analyze it so we can understand our users. We create reports to inform the business. The data is imported into the staging area (database), then the data warehouse, then we go to reporting, analysis, recommendations. These are all jobs that need to be performed in order.

Before Job-Runner this was all scheduled by cron. If one job failed, the follow-up jobs would fail too. There was no good overview of running, completed and failed jobs, hard to quickly see what (if anything) went wrong. Cron could start a new job if the previous long-running job was still running. Not good.

Job-Runner is a centralized dashboard, an admin interface, a broadcaster for jobs, a RESTful interface. Then there are decentralized workers (job-runner-worker), and a centralized WebSocket server (job-runner-ws-service). Those two communicate with Job-Runner.

The dashboard is HTTP, the queue broadcaster is ZeroMQ.

A project has a job template, for example a python snippet with some variables that are filled in. A job is created from this template. The job is executed in a worker pool. The job may run multiple times, for example once every day. You can also start it manually from the dashboard.

Job-Runner is a Django project, with TastyPie for the RESTful interface. There is a long running management command (the publisher), called broadcast_queue. The worker subscribes to this publisher. It publishes events to ZeroMQ. gevent handles the event queue. The WS Server is a simple proxy between ZeroMQ and the WebSocket server.

We started in December last year. We looked around for other solutions, but they did not fit our needs. The authentication system of Django worked nicely for us. You can setup notification e-mail addresses for a job or pool. Via the admin interface you can schedule jobs at a specific time. If a previous job run has not finished yet, the new job instance will not yet be started.

We have used this for hundreds or even tens of thousands of jobs. We only have one normal cronjob left.

The documentation is at http://job-runner.readthedocs.org

Personal website: http://www.brocaar.com

Jan-Jaap Driessen - Sentry

I work at Mind District. We set up Sentry in our web stack, because we don't want to hunt around for stuff in lots of logs on several servers.

Tip: use UDP for sending the logs to the central server.

We have used this in Python, but are now also using it in javascript, as simple as this:

MD.logging.warn('bar');

Inspired by Strophe. We have a few lines of code for that. Ask me. We also use log4javascript, which is 128 kB and that is minified. In development you want to use the console, but in deployment you want to use Sentry. The code uses Raven to capture the logged message and send the message to the Sentry server.

The WSGI integration is in Zope3, let me know if you are interested.

Diederik van der Boor - 10 things to take care of when Open Sourcing your package

For example django-fluent-pages, developed by me.

• License: do not use the GPL, but the BSD or the Apache license, otherwise you cut your project off from other Python community modules. [For the record: I do not agree with this. Maurits]
• Please provide a README with a reason for using your project. Maybe a screenshot.
• A changelog please. I don't want to look at github commits to check if my bug got fixed in a specific version.
• Documentation. For example on http://readthedocs.org. Look into sphinx.ext.autodoc for automatically generating documentation from your code. Also graphviz and intersphinx.
• Add tests. A runtests.py that is enough to run your tests without needing to setup anything is a good idea. Look into Travis for automatically running tests on a server. tox is good for running tests locally.
• Use the communication channels. Keep people updated. Have a website, with documentation.
• Please provide an example project, so people can easily see how it works.
• Create a good setup.py.

See my github account: https://github.com/edoburu

I count eight things, not ten, so I apparently missed two. Well, two points from the audience follow.

Audience:

• If you start losing interest in your project, look for someone who can take over, preferably someone who has already created some pull requests. Also give people commit access to your repository if they have made some good pull requests.
• Upload your distributions to PyPI, not your own small server. And leave all versions online, otherwise some people will be very annoyed as they are using it on their live site.

Eric - Two Scoops of Django

Daniel Greenfeld and Audrey Roy are writing a book on best practices for Django development. It is in beta, but you can already buy it. It is very good, so please do. The beta is 14 dollars and then you get the final version for free.

Get it at https://django.2scoops.org

Jan Murre - Webapp2

I am active with Python, Zope, Plone, Django, since 1999. I am currently working for a company in Utrecht and before that Pareto.

Webapp2 is a lightweight Python web application framework. It is compatible with webapp, its predecessor, which is the original Google App Engine framework. Webapp2 has several improvements, for example better URI routing and WebOb. See the webapp2_extras package for more stuff. Google has adopted it since GAE SDK 1.6.0.

It has WSGI integration. You can choose your templating language: Jinja2, Mako, etcetera. Forms: wtform, formish, etcetera. Persistence: for AppEngine there is BigTable. Stand-alone you can again use whatever you like: SQLAlchemy, some noSQL, etc. Use a WSGI container, like uWSGI, mod_wsgi, paste. So you pick and choose your own stuff on top of the micro framework.

Google App Engine is Platform As A Service (PAAS). You upload your application there. Comparable in that respect with Heroku. Easy to build, easy to maintain, scales and load-balances automatically, you pay for what you are using. There are several quotas and limits. For some applications we only have to pay the minimum of two dollars a week for the professional version. There is also a free version, with low limits for bandwidth, memory. Languages supported are Python, Java, Go (experimental). BigTable is a noSQL storage, with queries, sorting, transaction, caching built in. URL fetching. E-mailing. There are task queues (think: Celery). Scheduled tasks (think: cron).

You start the application with a simple, plain text yaml configuration. Of course I use Buildout to setup my local development environment, where you have some local storage. The recipe to use is rod.recipe.appengine.

There are other Python (micro) frameworks that you can look at, for example Flask, Pyramid. You can use those on the Google App Engine too, but it takes a bit more setup.

Conclusions for Webapp2: small, but nice. Quite some batteries included. Good routing engine. Plays nice with Google App Engine.

Together with someone else I created http://mijnvoetbaltrainer.nl that uses this, in combination with Amazon for hosting the video material.

More info on webapp2: http://webapp-improved.appspot.com

Pepijn Vos - Pygments, ctags

Pygments: code colorizer. Ctags: allows you to jump to a definition of some code. You can combine that. I did that in spelunking. You can use it to browse github repositories and click around in it. It downloads the repository and runs Pygments and ctags on it and makes it available.

Code: https://github.com/pepijndevos/spelunking

Demo server: http://raspi.pepijndevos.nl

Jan-Jaap Driessen - video conferencing

We have built some cool stuff for health care professionals who visit other people. It can be handy to do this via video communication. We did not base this on Skype, but on Google video conferencing. http://videmo.minddistrict.com There was a SOAP api that we had to interact with. If you have to do that, you should use suds. Using bosh, jabber, xmpp on the server. You can use it simply with an Android phone, no need to buy special hardware.