Safety and exactness of restore

• When restoring, first run checks for all filestorages and blobstorages. When one of the backups is missing, we quit with an error. This avoids restoring a filestorage and then getting into trouble due to a missing blobstorage backup.
• When restoring to a specific date, find the first blob backup at or before the specified date. Otherwise fail. The repozo script does the same. We used to pick the first blob backup after the specified date, because we assumed that the user would specify the exact date that is in the filestorage backup. Note that the timestamp of the filestorage and blobstorage backups may be a few seconds or minutes apart. So now the user should pick the date of the blob backup or slightly later. This date will give the same result with 3.1 and 4.0. But: when you use the new blob_timestamps == true option, these dates are the same.

Blob timestamps

• Added blob_timestamps option. Default is false. By default we create blobstorage.0. The next time, we rotate this to blobstorage.1 and create a new blobstorage.0. With blob_timestamps = true, we create stable directory names that we do not rotate. They get a timestamp, just like the repozo backup. For example: blobstorage.1972-12-25-01-02-03.
• When backing up a blobstorage, use the timestamp of the latest filestorage backup. If a blob backup with that name is already there, then there were no database changes, so we do not make a backup.
• Automatically remove old blobs backups that have no corresponding filestorage backup. We compare the timestamp of the oldest filestorage backup with the timestamps of the blob backups. This can be the name, if you use blob_timestamps = true, or the modification date of the blob backup. This means that the keep_blob_days option is ignored, unless you use only_blobs = true.
• Note: it is fine to switch to blob_timestamps even when you already have 'old' backups. Restoring those will still work.
• blob_timestamps = true may become the new default later (maybe 4.1). This may even become the only valid value later (maybe 5.0), removing the creation of blobstorage.0. This would simplify the code. If you don't like this, please speak up and create an issue.

Archiving and compressing blobs

• Renamed gzip_blob option to archive_blob. Kept the old name as alias for backwards compatibility. This makes room for letting this create an archive without zipping it.
• Added compress_blob option. Default is false. This is only used when the archive_blob option is true. When switched on, it will compress the archive, resulting in a .tar.gz instead of a tar file. When restoring, we always look for both compressed and normal archives. We used to always compress them, but in most cases it hardly decreases the size and it takes a long time anyway. I have seen archiving take 15 seconds, and compressing take an additional 45 seconds. The result was an archive of 5.0 GB instead of 5.1 GB.
• Note that with both archive_blob and blob_timestamps set to true, you get filenames like blobstorage.1972-12-25-01-02-03.tar.
• Added incremental_blobs option. This creates tarballs with only the changes compared to the previous blob backups. This option is ignored when the archive_blob option is false.

Various

• No longer create the fullbackup script by default. You can still enable it by setting enable_fullbackup to true.
• Added Python 3 support. The integration with plone.recipe.zope2instance is not tested there, because there is no Python 3 compatible release of it yet.

Upgrading

• In most cases you can simply use the new version without changes.
• Adding blob_timestamps = true is highly recommended. If you do this, you can remove the keep_blob_days option, unless you use only_blobs = true.
• If you want the fullbackup script, enable it by setting enable_fullbackup to true.
• When you used the gzip_blob option, you should rename this to archive_blob. Maybe enable the compress_blob option, but you are probably better off without this.

Wouter van Atteveldt: Large scale search and text analysis with Python, Elastic, Celery and a bit of R

I got my PhD in artificial intelligence. I am now a social scientist.

Why do you need text analysis? Example. In 2009 Israel and Palestina were fighting. How did the media report? I downloaded lots of articles and did text analysis:

• U.S. media: lots about Hamas and their rocket attacks that provoked Israel.
• Chinese media: more about the Gaza strip, actions on the ground by Israel.

So you can learn from text analysis. There is a flood of digital information, and you cannot read it all, so you use text analysis to get an overview. You need to go from text to structured data.

Facebook research. They changed the timelines of 600,000 people, some were shown more positive messages, some more negative. More positive messages resulted in less negative messages written by those people, but also less positive messages [if I saw it right]. Lots of things wrong with this research.

We can do much better, with Python. Python is platform independent, easy to learn. Half the data science community is in Python.

Or we can do stuff with the R language. It is aimed at statistics and data. There is convergence via numpy and pandas and R packages.

I helped create https://amcat.nl/ where you can search data, for example finding articles about Dutch politics in the Telegraaf. You can search there for 'excellentie', and see that this Dutch polite term for ministers was used until the sixties, and then resurfaced recently when a minister wanted to reintroduce this, and got satirical comments.

AmCat is a text search front end and an API, written in Python and R scripts and ElasticSearch. We upload texts to elastic on two separate nodes. You can never change an article: they are unique and hashed, so you get the same results when you search again next year.

For the Telegraaf paper, there is no data between 1995 and 1998. The previous years were digitised by a library, and the later years by Telegraaf themselves, but the current owner is not interested in filling the gap.

Our script interface is a Django form plus a 'run' method. This means our CLI, HTTP and API front end can use the same code.

For async jobs we use Celery. I know a professor who likes to write a query of several pages, which can take more than an hour to handle, so he would normally get a timeout. So we do that asynchronously.

We want to do NLP (Natural Language Processing). Or preprocessing. There are many good tools, like Stanford CoreNLP for English. We developed NLPipe, a simple NLP job manager. It caches results. Separated into server and workers. Workers can run on AWS via Docker, but since we have no money we are looking at Surf HPC but they don't support Docker, so we look at Singularity instead. Experts welcome.

Reinout van Rees: Fast querying and filtering with Django

[Same talk as on PyGrunn this year. I copied my summary and extended it.]

Goal: show what is possible. Everything is in the Django documentation. Just remember a few things you see here. If you know it is available, you can look it up.]

The example case I will use is a time registration system. Everyone seems to do this. Oh, less hands here than at PyGrunn. The tables we will use are person, group, project and booking. A Person belongs to a Group. A Booking belongs to a Project and a Person.

The Django ORM gives you a mapping between the database and Python. You should not write your own SQL: Django writes pretty well optimised SQL.

Show all objects:

from trs.models import Person, Project, Booking
Person.objects.all()

Basic filtering:

Person.objects.filter(group=1)

specific name:

Person.objects.filter(group__name='Systemen')

case insensitive searching for part of a name:

Person.objects.filter(name__icontains='reinout')

or part of a group name:

Person.objects.filter(group__name__icontains='onderhoud')

name starting with:

Person.objects.filter(name__startswith='Reinout')

without group:

Person.objects.filter(group__isnull=True)
Person.objects.exclude(group__isnull=False)

Filtering strategy:

• sometimes .exclude() is easier, the reverse of filter
• you can stack: .filter().filter().filter()
• query sets are lazy: only really executed at the moment you need it. You can use that for readability: just assign the query to a variable, to make complicated queries more understandable
• start with the model you want

Speed:

This will use one initial query and then one extra query for each person:

systems = Person.objects.filter(group__name='Systemen')
for person in systems:
    print(person.name, person,group.name)

Not handy. Instead use the following.

• select_related: does a big join in SQL so you get one set of results:

  for person in systems.select_related('group'):
      print(person.name, person,group.name)
  

  This does one query.

• prefetch_related: does one query for one table, and then one query to get all related items:

  for person in systems.prefetch_related('group'):
      print(person.name, person,group.name)
  

  This does two queries. Both can be good, depending on the situation.

• It is expensive to instantiate a model. If you need only one or two fields, Django can give you a plain dictionary or list instead.

Dictionary:

systems.values('name', 'group__name')

List of several fields in tuples:

systems.values_list('name', 'group__name')

Single list of values for a single field:

systems.values_list('group__name', flat=True)

Annotation and aggregation:

• annotate: sum, count, avg
• aggregation
• groupby via values (bit of a weird syntax)

Aggregation gives totals:

from django.db.models import Sum
relevant_persons = Booking.objects.filter(
    booked_by__group__name='Systemen')
relevant_persons.aggregate(Sum('hours'))

Annotation adds extra info to each result row:

relevant_persons.annotate(
  Sum('bookings__hours'))[10].bookings__hour__sum

Filter on bookings for maternity leave, group bookings by year, give sums:

Booking.objects.filter(
    booked_on__description__icontains='ouderschap'
).values('booked_by__name', 'year_week__year'
).annotate(Sum('hours'))

Note: I used the faker library to replace the names of actual coworkers in my database with random other names.

Practice this with your own code and data! You'll get the hang of it and get to know your data and it is fun.

What I hope you take away from here:

• Read the docs, you now have the overview.
• Make your queries readable.
• Practice, practice, practice.

[I will toss something extra in from PyGrunn, which was probably a question from the audience.]

If you need to do special queries, you can create a sub query yourself:

from django.db.models import Q
query = Q(group__name='Systemen')
Person.objects.filter(query)

You can write filters that way that are not in default Django.

Twitter: @reinoutvanrees

Wojtek Burakiewicz: Building robust command line tools with click and flask (and microservices)

I worked at Byte and am now working at Optiver, in the tooling team, using Python for data center management. I use Flask and Click, made by the same people. Our sd tool is internal, it is not on the Internet. It deploys Python applications to servers with virtualenv.

On the server side we us stash, artifactory, ansible+Jenkins, supervisord, JIRA. On the user side we have our sd tool to talk to the programs on the server.

You would normally need passwords on some servers, auth keys or ssh keys on others. Some ports open, a different one for each new app. Messy.

So sd talks to an api that sits in between. The api then handles the messy talking to the servers, and you store authentication and port numbers in a central config, instead of on the computer of each user. All users talk the the api, the api talks to the servers.

For the server side api we use flask and flask_restful. For the client side we use click.

When you install Django, you get a house. When you install Flask, you get a brick. Sometimes you want a house. Other times all you need is a brick. You can build something entirely different with a brick. It is easy.

Making an api is done easily with:

from flask_restful import Api, Resource

Then click for the command line tool. I always struggle with argparse, but I like working with click:

import click

@click.command()
@click.option('--count', default=1, help='...')

Click makes you structure your application in a nice way:

tool/
  cli.py
  actions/
    action1.py
    action2.py

We use a trick to force users to upgrade. With every request to the api we send the version of our cli. The api checks this against a minimum version and aborts if the version is too old.

Documentation:

• Flask: http://flask.pocoo.org
• Click: http://click.pocoo.org