See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.

I am member of the operations team at Paylogic. Responsible for deployments and support.

Handling taks that are too lengthy for a request/response loop.

Option 1: Celery plus RabbitMQ as backend. Widely used, relatively easy. RabbitMQ proved unreliable for us. Our two queues in two data centres would go out of sync and you needed to reset the queue: manual work and you lose data. This is a known problem. Also, queues could get clogged: data going in but not out.

Option 2: Celery plus MySQL as backend. So no separate queue, just our already running database. But it was not production ready, not maintained, buggy.

Option 3: Gearman (MySQL). Python bindings were again buggy. Could run only one daemon at a time. By default no result store.

Option 4: do it yourself. Generally not a great idea, but it does offer a "perfect fit". We built a minimal prototype, which grew into Taskman.

MySQL as backend: may be fine, but it was not a natural fit. "Thou should not use thy database as a task queue." Polling: there is no built-in event system, though there are hacks that pretend this. Lock contention between tasks is a bit hard. Some options. You can enable autocommit, so you do not need a separate commit. No SELECT FOR UPDATE but simply SELECT. Fuzzy delays. Data growth: queue can grow with time, but you can remove old data.

Task server, daemon with Python and supervisor. Loop: claim task from the database, spawn, sleep, repeat.

Task runner is a separate process. Sets up Python environment where the task runs, it runs the task, does post-mortem: get results, report back.

Task record (database row) simplified: function, positional and keyword arguments, environment, version of environment, state (pre-run, running, ended/finished, ended/failed), return_value.

Task server is an independent application. It does not know about the application that is actually running the tests. Applications need to register to the server with a plugin: methods get_name, get_version, get_environment_variables, get_python_path. Result of task must be a json string.

A task can report progress, by accessing its own instance. So it can say '20% done.' Tasks can be aborted. Task start time can be constrained, e.g. if the task is not started within ten minutes, delete it because this one is not useful anymore. Exception handling.

Taskman properties are optimized for long running tasks, not for minor tasks. Designed for reliability. Tasks will not get lost at some point, they will get executed, and get executed only once. It is less suited for a blizzard of tasks, lots of small tasks, more for heavy database processing. There is no management interface yet. If you must, you can use phpmyadmin currently...

I really want it to be open source, but we have not gotten around to it yet. We first want to add package documentation on how to set it up.

See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.

I work at MindDistrict. Today I will talk about a web service I built with Tornado. It needs to talk to lots of things, without doing much computation itself. It uses Python 3, because it was about time to use it.

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado is IO loop based. Does not work with a WSGI server. It is easy to implement (micro) web services.

What is an IO loop? Old programming technique, from when threads did not exist or were expensive. It does not use threads, only one flow to execute the code. The loop calls you, you do things, when you are done you relinquish control to the loop. Not efficient for computation intensive tasks.

You can use callbacks, but they are ugly. With generators for the IO loop you can replace them. With @tornado.gen.coroutine you decorate a function and let it yield results.

It works with 'future' objects: a result that still needs to be computed. Comparable with promises in Javascript. You can create them yourself:

def some_function():
    future = tornado.concurrent.Future()
    # do stuff
    future.set_result(42)
    return future

Tornado provides an improved version of subprocess.pOpen. Create a future, call the subprocess, when it is done set the return value or an exception.

You cannot use normal threads or file locks. But you can implement a lock mechanism with futures.

Future objects are reusable for caching. 'memoize' type decorators work.

concurrent.futures is part of the standard library in Python 3. With it you can run your heavy computations in a separate process. It works with tornado: create a coroutine and use the Python 3 futures in there. There is a backport for Python 2.7.

Then Python 3. The syntax for generators is much better. Enough support in main packages. It works nice for us in production.

See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.

I am going to talk about modelling. Process modelling and data modelling. My colleague Henk Doornbos will give a more detailed technical talk after this.

Many ICT systems do not work, a lot of projects fail. Dutch government loses about 5 billion euros a year on failed projects. Expected functions are not provided, and offered functionality is not wanted. For example medical patient information systems.

Main cause of failure: the initial design does not match the end-user requirements.

Possible solution: model driven approach (MDA). You can see models as very high level programming constructs. You might think of UML. But let's revisit this.

Wittgenstein: Don't ask for meaning (of data) but of its use.

Our process: start from business process models (our context) and derive data models, mostly automatic.

Validate your models. The only real correctness is end-user correctness. Capture the main requirements from experts. Anyone can write a model, but how do you know it is correct. UML class diagrams are hard to read and write. Instead, we use a structured natural language that is easy to validate by domain experts.

Are the requirements consistent and complete? Go through it in cycles. Do they converge to a stable model? Models are assumptions about how the domain expert would like to do her job. Try to invalidate the model, hunt for false assumptions.

We use business process modeling and notation. Many modeling primitives, of which we have chosen a subset of eight. Process is a collection of pools. Pool has swim lanes for each stakeholder.

Clients learn the notation pretty quickly. We stick as close as possible to the domain expert's language, which may just be Dutch.

High level overview of a process flow:

• You start
• code the input
• process/transform the input
• decode the output
• stop.

Fact-based modeling approach. Get data. We talk about objects and facts. Facts are sentences that are true or false. The European Space Agency uses this approach for their satellites.

Our flavor of that approach is OLE: Object and Logic-based English. Structured natural language. Better suited than graphical models for validation by domain experts. It is very precise and expressive. OLE can be compiled a prototype to pyDatalog for example. From the site: pyDatalog adds the logic programming (think: the prolog language) paradigm to Python's extensive toolbox, in a pythonic way. More about that in the next talk in this room.

An activity may be coded as for example six statements. Activity A is on Date D. Start Event E leads to Statement S. Entity Type T is identified by Number N. Things like that.

We are using this for a contract system. Contracts are built from small initial building blocks. They are combined recursively into a kind of algebra for contracts. That is turned into a pyDatalog program.

The data modeling part is difficult when you are not used to it, but clients can come up with activities themselves and will spot errors in the logic. They may find out that their own process is more difficult than they previously thought. Or they discover that they do it differently than their colleagues.

How good is this when the process later needs to be adapted? We identify which part of the process needs a change. Change the model, compile that part again.

See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.

At RedHat we try to be newbie friendly, because everyone is a newbie in lots of contexts. You specialize in very few fields.

Declarative knowledge: Amsterdam is the capitol of the Netherlands, A is the first letter, etcetera.

Procedural knowledge: how do I get to Groningen, how do I train for a marathon, how do I learn a new language.

So: what versus how.

Example: when you start learning a new programming language, you may want to stick to your trusty editor and version control system instead of switching those at the same time.

I used to dance, had a degree. You do not earn a lot of money that way. I learned languages. Later html, css. I applied, but I usually not even got an interview, because I had a degree in dancing. So I got a degree in computers. I got hired by RedHat.

At RedHat we say "drink from the firehose:" inhale something that is overwhelming. Go with the flow. You have 90 days to pass the exam. It is like a boot camp. It is not about what you do exactly, but how you manage it: do you drown and give up, or do you battle on. So you learn lots of new technologies and after you pass the exam, you apply that knowledge. Not the knowledge about those new technologies, but about how you learned them. You just keep on learning new stuff, and are not afraid of it. No pressure...

For a workshop I setup this Django Girls blog: http://leanderthalblog.herokuapp.com

Challenge: deploy to OpenShift. http://django-leanderthal.rhcloud.com So, it is deployed, but I ran into problems, so you don't really see anything on it. Battling on, challenge still accepted.

Figure out who you are, what you are good at. What programming languages did you learn? How did you do that? Apply that knowledge to new programming languages. Did you learn via tutorials? Readme? Asking others? Reading books, blogs? Just do it and practice, practice, practice? Did it work or not? Apply that knowledge.

On 19 September 2015, there is a Django Girls workshop right here in Groningen. See http://djangogirls.org/groningen/ Free programming workshop for women. We need coaches. Men are welcome too.

On Twitter: rainsdance