Weblog
Oleg Pidsadnyi - Behaviour driven design with PyTest
Oleg Pidsadnyi talks about behaviour driven design with PyTest, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
Code of pytest-bdd: https://github.com/olegpidsadnyi/pytest-bdd
I will talk about behaviour driven development in Python. What is it? You define a scenario with strict language, like given this and that, expect this. It is readable for both programmers and business logic persons.
There are several ways to do this. You can use Lettuce or Freshen, plus Splinter. But imperative coding style does not work well here, I think. Given I have two books. How do you do that? context.books = [...]? context.book1, context.book2? pytest-bdd has the concept of expecting and returning values, with @pytest.fixture:
@pytest.fixture def author(): return Author() @pytest.fixture def book(author): return Book(author=author) @given('I have two books') def article(author): return [book(author=author), book(author=author)] @given('I have an article') def article(author): return create_test_article(author=author)
This is much more explicit.
It can do browser tests. You can use a normal browser, like Firefox, or you can set it up headless with phantomjs.
pytest-bdd is inspired by the Robot Framework, but we had some different requirements.
Douwe van der Meij and Brandon Tilstra - MVC revisited with Diazo
Douwe van der Meij and Brandon Tilstra talk about MVC revisited with Diazo, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
We work for Goldmund, Wyldebeast and Wunderliebe, the sponsors of this conference with the longest name.
We use Diazo for theming. Your application creates an html page. Your designer creates a standard html template including css. With Diazo you merge the two. It separates the content creation from the styling.
The technique is plain old XSL transformations. Nothing new.
Why should you use it? Designers and developers look at a certain project in a different way. The designer sees the beautiful outside of the car and the developer sees the gritty details under the hood. Developers usually want to stick to developing features in a minimal design, just some standard html preferably without any css.
If you look at the Model-View-Controller paradigm with Diazo in mind, it makes the View part easier: the developer handles the application part and the designer handles the styling part. The designer does not need to know Django or Plone templates.
Brandon is busy making Diazo available for other applications than just Plone. For Plone a tool is available: plone.app.theming. Since Plone 4.3 you can edit the theme inside Plone. A designer can do that, with a WYSIWYG editor in Plone and a developer can tweak the code with a code editor in Plone.
Brandon is working on thememapper, a standalone theme editor, written in Python. thememapper.core is the tool itself. thememapper.diazo is a diazo server to be used with thememapper.core.
Question: Aren't you trying to solve an organizational problem? Shouldn't the designer and developer be talking to each other?
Answer: We often get a design in Photoshop, give it to a third party front-end party that creates html and css from it, and we as developers create the Diazo rules.
Alessandro Molina - High Performance Web Applications with Python and TurboGears
Alessandro Molina talks about High Performance Web Applications with Python and TurboGears, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
I am @__amol__ on Twitter. I am a member of the TurboGears team. I will talk about general rules which apply to any framework, some quick wins for TurboGears, some real cases, my personal experiences and preferences, feel free to disagree.
People seem obsessed with the raw speed of web servers. But you are not going to serve a "Hello world" page. My personal stack has nginx plus mod_wsgi or nginx plus Circus-Chaussette-gevent.
Try to avoid making everything an asynchronous (AJAX) request. Browsers have limited concurrency. HTPP has overhead. You will actually slow things down if you have too much of them. Your page may start fast but complete slow. Learn your framework and how you can use it the best way for your use case.
TurboGears is a framework for rapid development, encouraging flexibility. It was created in 2005, with 2.0 a major rewrite in 2009 to embrace the WSGI standard. It is based on object dispatching. You can use regular expressions for url matching, but they can get messy, so write them only when you must. By default an XML template engine with error detection. Declarative models with a transactional unit of work. It has built-in validation, authentication, authorization, caching, sessions, migration, etcetera.
Features versus speed: TurboGears is a full-stack framework. That makes it quite slow by default. You can switch things off that you do not need. The team invested effort to constantly speed it up since the 2.1 release. Keeping all the features around has its price, obviously. To cope with this, a minimal mode got introduced, that switches several things off that have the biggest influence on performance. You go from about 900 to 2100 requests per second.
Avoid serving static files in your framework. Let some other part of your stack handle this. This can take a lot of load from your application server.
Use caching. Caching means preorganizing your data the way you are going to use it. For example with a NoSQL database you can load the comments directly when accessing the page, so you don't need to load them separately. Frameworks usually provide various types of caching. Get to know them and use them.
Use HTML5 and Javascript. Invalidating your whole cache just to add a message "Welcome back, mister X" is not a good idea. Cache the result and use Javascript to do minor changes. If you are using varnish, nginx or any other frontend cache, consider using Javascript plus localstorage instead of cookies for trivial customizations, because cookies disrupt the cache.
Cache the result of rendering a template, with a cache key.
Entity caching: cache parts of your page, for example the html of one comment or notification.
Proactively update the page in your cache: when you edit the page, update the cache before your visitors ask for it.
If you are struggling too much with improving performance, you are probably doing something your application is not meant to do. Also, football fans are really eager for updates.
Offload work. Update the core cache to provide the author with an immediate feedback. Let some other process, program or thread handle the related background changes. For example, use something like Celery.
New Relic App Speed Index reports an average of 5.0 seconds of response time for accepted experience: http://newrelic.com/ASI If response time is less than 200 milliseconds, this is seen as 'right now'.
Sorry, there is no silver bullet for speeding up your application.
Kenneth Reitz - Python for humans
Kenneth Reitz talks about Python for humans, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
Follow me on twitter: @kennethreitz_. I work at Heroku. I am a member of the Python software foundation. I created the requests library that makes is easier for humans to retrieve a web page in Python.
I want everything I build to be open source. I build for open source, pretend it is open source even when it will never be released to the public because it is too client specific. Add documentation, please, also to your internal tools.
Open your Python prompt and see the Zen of Python:
>>> import this
Beautiful is better than ugly: you don't need lots of curly braces in Python. Explicit is better than implicit.
Most importantly for today: there should be one - and preferably only one - obvious way to do it.
Welcome to paradise. You have found Python, a language with a beautiful zen philosophy and all is going to be fine. Lies!
Look at some Ruby code to download the contents of an https web page. Perhaps a bit too many lines, but it is pretty straight forward. Now you switch to Python. Which library? urllib, urllib2, something else? Okay, you use urllib2 and then you have to add some password manager and lots more lines that are unclear. You will leave and never come back.
This is a serious problem. I think HTTP should be as simple as a print statement. It is used so often! Our world is connected over HTTP right now so we need this to be simpler.
Python needs more Pragmatic Packages. Deal with things sensibly and realistically, in a way that is based on practical rather than theoretical considerations. Make it practical for humans, for the common case.
The requests library is HTTP for humans. A small set of methods with consistent parameters. Do this for more modules! Fit the ninety percent use case. Features, efficiency, performance, etcetera are important, but you should ignore them for such a practical package. Go for the common case. Write the README the way you think it is supposed to work.
"Cool story, bro, but why should I do with this?" It's worth your time and everyone else's time as a developer.
Some things that could be improved:
- File and system operations. We have the modules, sys, shutils, os, os.path, io. Which should you use for a task? It is really difficult to run external commands and this blocks dev+ops folks from using Python.
- For installing Python there are various ways. Use the Python that came with the system or compile your own? Python 2 or 3?
- XML hell. etree annoys people. lxml is awesome, but can be difficult to install.
- Packaging and dependencies. Pip or easy_install? How about an easy_uninstall? Distribute or setuptools? [Holger Krekel will talk about this topic later today in the keynote.]
- Dates, datetimes. Which module: datetime, date, time, calendar, dateutil, version 1.5? Timezones, they are ridiculous.
- Unicode. We'll just skip that one.
- Testing. Lots of different ways to create tests. Doctests and unit tests. Various test runner libraries, like nose and tox.
- Installing dependencies. Python-mysql, if you remember the exact name. (My solution: just use Postgres.) Python Imaging Library needs several system packages before you can install it. mod_wsgi: if you install this, which Python version are you using, the one from the system?
Hitchhiker's guide to Python: http://python-guide.org The goal of this project is to write down the tribal logic, documenting best practices. A guide book for newcomers. A reference manual for seasoned pros. It tries to document the one - and preferably only one - obvious way to do it. Please contribute documentation here. This lets of practice what we preach.
Lets fix our APIs and improve our documentation.
requests might get into core at some point.
requests started as a wrapper around urllib2. It was a nightmare. It now uses http to handle the lower level parts.
If I would have said 'yes' to all feature proposals for requests, it would have been too much. People can write a library around requests. Saying 'no' made it possible to create and maintain a good architecture.
Python Users Netherlands meeting 18 April 2013
Summary of the meeting of the Dutch Python Users group on 18 April 2013.
We were welcome at the offices of Fox-IT in Delft for presentations, lightning talks and meeting fellow-Pythonistas.
(Note: my brother Reinout also made a summary.)
Reinout van Rees - Report on testing
Testing is good, but not everyone does it. How about making a report of who is actually writing tests and which project has commits with tests? With githubinfo you get a report of your projects on github sorted by the amount of commits that have tests in them. I send an email out to my colleagues with that info every week. The email also contains a list of committers sorted by the amount of commits with tests. This should help in getting people more aware of testing. You can configure it with a json file.
https://github.com/nens/githubinfo http://pypi.python.org/pypi/githubinfo
Ronald Evers - Dev'ing at Fox, how do they do it
I will show you a little bit about how we do development and operations here at Fox-IT. I am working on DetACT, fraud detection software for online banking. We try to detect fraud in the data stream. Low-level in C++, the rest in Python. We have sold DetACT to several banks.
We switched from subversion to git. We use Gerrit. This is a code review system. Before you are allowed to push to the master branch, you need to get two upvotes for your commits, which is done on the Gerrit server. You can compare it a bit with a pull request on github.
We use Jenkins for automated tests. This also works nicely in combination with Gerrit.
We have to respond quickly to new security attacks, so we deploy as often as possible. We use irc, Fabric, Sentry, Munin, Nagios, request tracker. Sentry is an exception tracker: what is going wrong on which server? We have built a simple irc bot for Gerrit that tells us on an irc channel what is going on with the code.
For writing an irc bot: https://github.com/ronaldevers/botlerplate
Alternatively: https://github.com/github/hubot
With Fabric we execute actions on several servers. It is like Make, but also works on remote servers. Selecting which hosts you want to deploy on is actually the hardest part for us. So we have built something for that. We change the globals() dynamically to add extra targets. It works.
JSON, HTTP and the requests library are your friends. Connect everything. Automate everything. Monitor everything.
Yes, we are hiring.
Jeroen Vloothuis - Get more attention for your project
Do not let your open source work lead to nothing. Or: how to let your project get the deserved attention.
A lot of people do open source projects, but they make a few mistakes in communication and the project does not succeed. So a few simple marketing tips to help you maximize your user audience.
People care about themselves. In documentation talk to the user: "This software lets you write efficient code." Get your users engaged. Emphasize what it does for them.
"Your project solves my pain, right?" That is what the user wants to know. Why would you write software when not to solve a real problem? "Django makes it easier to build better web apps more quickly with less code." Sounds good. Compare that with: "Flask is based on Werkzeug and Jinja2 and has a BSD license." Yeah? So? What can I actually do with it?
So: why did you start your project and what problems does it solve?
Talk about benefits, not features. "My project has a FrobNob 2000." Right. "My project saves you time." That could be useful. "Rely on MySQL to save time and money." Good. "We are blah compliant, have full support for blah." Excuse me? If you are ACID compliant, say something like "we will not corrupt your data."
So: focus on benefits. Features just support your claim.
A picture is worth a thousand words. In the case of Python the picture can be a few lines of source code, showing how easy it is to use your package. Show, don't tell.
So focus on:
- your users
- their pains
- benefits first
- pictures.
Question: Won't developers like the facts more? They do want to know if a database is ACID compliant.
Answer: Fine, but still start with telling that in simple terms.
Jasper Spaans - A talk about nothing
This is a lightning talk about nothing. I counted the zeroes and ones in the DetACT project. The zeroes clearly win, with almost sixty percent.
See https://github.com/jap/bittally
PEP-336: make None a callable object. That would be handy, as that would mean you can do None.endswith('wawawa') and it won't fail so you need less code. Guido denied this pep. I wrote this code if you want to use it anyway.
Onno Broekmans - Programming scientists
PhD student at University of Amsterdam. I am wondering who you guys (and girls!) are. Lots of web development here, I think. I am working as a scientist. Scientists have invented the atom bomb, particle accelerator, and we do it with spaghetti code. Most scientists have no background in programming. We teach it to ourselves. We mess around a bit. What is version control? Bug tracking? Unit testing? Never heard of it.
But we are supposed to cure cancer and figure out this climate thing. Scientists should be educated in programming and the activities around it. This is not easy to do, but we are working on it. In Canada there are a few guys that do Software Carpentry. They are coming to the Netherlands. If you are a scientist, please join them. If you have ideas about how to teach people about Python, I would be interested to hear about that. Are you interested in being involved, talk to me.
Not many code from scientists makes it to the community. We should, but we don't. It is the elephant in the room.
Jan-Jaap Driessen - Fanstatic 1.0a
fanstatic takes the pain out of static resources. Your cool web application does a lot of difficult computations and a lot of dumb stuff: serving Javascript and css files.
Fanstatic helps you there, with lots of js.* packages on http://pypi.python.org. You can do your version requirements for javascript files in the setup.py of your python package.
You can bundle resources into one. Since last night it has compilation support, so coffeescript, sass, less, clojure, etcetera. Fanstatic saves you time. So if you write sass you no longer need to compile it yourself, as fanstatic does it for you.
You no longer need to think about getting all your javascript dependencies in the html page. You let Python and fanstatic handle it.
Did you see me compile anything during this demo, and bundle those compiled resources, and minify those bundled resources? No! I save so much time and have so much less pain! Fanstatic saves the day.
You may not want to install all the compilers and minifiers on your server and fanstatic can help you there too.
It works with WSGI (fanstatic is the middleware), Zope, flash, Django.
Bundling is a hard topic. It depends on your stack and your setup.
Question: when would you not use fanstatic?
Answer: when I have a lot of time.
We support injection in the html head or at the bottom if your resource can handle that.
There is a branch for loading resources from a content delivery network.