I see a bit too often that some kind soul adds Plone 4.1 compatibility to a package and then does not notice that Plone 3 compatibility is destroyed. In at least one case this is totally not needed and easily avoided.
The problem that people try to solve is that on Plone 4.1 a lot of packages will need to explicitly load the permissions.zcml file of Products.CMFCore. That could be done like this, but this is bad:
<!-- BAD CODE: this breaks Plone 3 compatibility! --> <include package="Products.CMFCore" file="permissions.zcml" />
The problem with this code is that it breaks Plone 3 compatibility: the Products.CMFCore version used in Plone 3 does not have a permissions.zcml file! You will get an error like this when you try to startup your Plone 3 site:
IOError: [Errno 2] No such file or directory: '.../Products.CMFCore-2.1.2-py2.4.egg/Products/CMFCore/permissions.zcml'
Note that on Plone 4.0 the loading of this file works fine but is not needed, so avoiding it is probably very slightly better in startup time. No big deal though.
The proper way is to load this file conditionally. This is easy enough, with just two extra lines:
<!-- Good code: this keeps Plone 3 compatibility; thanks! -->
<include package="Products.CMFCore" file="permissions.zcml"
xmlns:zcml="http://namespaces.zope.org/zcml"
zcml:condition="have plone-41" />
Of course if you know that for other reasons your package is not Plone 3 compatible then you could safely omit the condition. But it does not hurt to keep it for clarity.
All this is shown in the Plone Upgrade Guide at http://plone.org/upgrade, specifically here.
Note that the plone-41 definition that is used in the zcml:condition is defined in the meta.zcml file of Products.CMFPlone 4.1.3 as follows:
<configure
xmlns="http://namespaces.zope.org/zope"
xmlns:meta="http://namespaces.zope.org/meta">
<meta:provides feature="plone-4" />
<meta:provides feature="plone-41" />
</configure>
So if you think you could use something similar in your code, this is where to start looking.
Meanwhile, loaded with this knowledge you can participate safely in the Plone TuneUp this Friday 16 December 2011, and update some packages for Plone 4. Thanks!
This month's PUN was hosted by Minddistrict, which has acquired The Health Agency (THA). Jan-Jaap Driessen introduced the companies. They build online tools for health care. Are you looking for a position of 1, 2 or 3 years at the Leuphana University (close to Hamburg, Germany)? They have a spot available there.
- Jan-Jaap Driessen: Connecting the bots
- Boaz Leskes: PyCounters
- Steve Alexander: Celebration, Motivation, Distribution
- Frank Niessink: Ran 160242 tests in 13.716s
- Vincent Pretre: collective.autosaveform
- Sylvain Viollon: context managers and generators to work on lots of objects
- Remco Wendt: showing two python tools
Jan-Jaap Driessen: Connecting the bots
Now for something more technical. We use buildbot to test our software. We use Selenium (using gocept.selenium) to test in a real web browser, including javascript. Instead of checking the end result of clicking with Selenium by going to a page, you can also check the database directly. We have a local browser farm, so with several versions of several browsers on several operating systems. You can try setting up something yourself or use the saucelabs paid service for this. So we have this setup:
zc.buildout + buildbot + selenium = nightly testing on all browsers
When this is running on saucelabs you can actually see the screen of the browser that is being tested at that moment! You can also see a video of it later and share that with your client to show that your software is working or that this particular bug has been fixed. The saucelabs service is pretty cheap actually, cheaper than setting up some hosts in the cloud yourself and keeping them running.
Boaz Leskes: PyCounters
PyCounters is a light weight library to monitor performance and events in production systems. We started this project because we had a performance problem. We had lots of layers in the system, so it was challenging to see where the bottleneck is. PyCounters gives us that. It is a combination of light weight events, counters to collect needed statistics, and reporters to regularly output statistics. Short example to see how many times per second a function is being called:
from pycounters.shortcuts import frequency
@frequency("req_per_sec")
def serve_a_request():
"""do something interesting and serve a request"""
pass
Some possibilities:
- We can calculate frequencies, average values per time, simply counting.
- There is JSON file output.
- A helper class for Munin plugins.
- Support for counting what happens in a single thread or for using multiple processes or servers.
Future plans:
- Using context managers (the with statement) is now planned, as decorators are not always fine grained enough.
- Django specific counters.
- More output formats (suggestions welcome).
See documentation at http://pycounters.readthedocs.org/ and source code at https://bitbucket.org/bleskes/pycounters
Note from Maurits: do not confuse this with the pycounters module of François Beausoleil which does something different, though it seems to have overlapping functionality.
Steve Alexander: Celebration, Motivation, Distribution
"Now we're talking." I was a python programmer, then I worked for Launchpad, then Canonical (making Ubuntu), then I did acting classes and stuff. Then I started a company to talk about how we work. We have really complicated brains. I am using a metaphor: the hero's journey. George Lucas used that same story to create the Star Wars movies. When we go to work we also go on a journey and we should enjoy it.
Some of us work alone, some in a community or team. Who is my team? What is my value to the team? We need to achieve a goal, for example fixing a bug or other issue somewhere in a tracker. You fix or create something and that means you add value to the team. That is the outer journey. There is also an inner journey, some kind of transformation that goes on. (Insert some Star Wars and Indiana Jones jokes here.) You have some personal 'Aha!' moments, where you discover amazing things, or that you can do amazing things. What those moments tell you is your team's value to you.
I wrote a blog article about celebrating the hero's journey. See http://blog.nowtalking.nl/archives/55. Lots of different reactions are possible from colleagues or clients when you tell them "Hey, I fixed a bug."
Frank Niessink: Ran 160242 tests in 13.716s
Task Coach: your friendly task manager. This is a project that has been going on since 2005. Done in Python of course, and some Objective C. It is open source. Runs on Windows, Mac OS X, Linux, BSD, iPhone. We try to release often; last year about twice a month. We have quite some downloads, in total at least 2 million downloads. More than 4000 users use it daily. Over 4500 unit tests and with 40 languages over 160,000 language tests. Translatable strings can contain keyboard shortcuts; translating this wrongly (e.g. translating Ctrl into German as Strg instead of leaving it at Ctrl) can actually crash the program. We have generated a unit test for each translation, like checking whether a translator forgot to add ... when necessary or indeed checking for program crashes.
Vincent Pretre: collective.autosaveform
collective.autosaveform is an open source project for Plone. A user has entered a lot of data in a web form, his internet connection quits and he gets mad because he has lost everything. With collective.autosaveform we save the form every five seconds or every click. We use localstorage for this in the browser, supported by IE8 and higher and normal browsers. After ten local saves we also send the data to the server so in case the browser crashes or something like that the data can still be made available later. When something bad has happened and the user is loading the form again, we compare the local and remote database versions and use the most recent. You need to register the form for autosaving in Plone with a few lines, and use one jQuery line in your javascript. It's configurable, friendly on the server and for the user, supported on multiple browsers, supports all HTML inputs except file. It should be pretty doable to make it work in Django.
See the slides.
Sylvain Viollon: context managers and generators to work on lots of objects
This is code that is hard to summarize, but you can do interesting things with context managers and generators, at least with Python 2.6 and higher. You can save some memory but more importantly you can save a lot of lines of code with good use of these utilities.
See his code at http://pastebin.com/hgqiSTVz. The long stretches of white space between the code parts should be seen as separating different versions of the code.
Remco Wendt: showing two python tools
vcprompt: show in the command line prompt which version control system (vcs) you are using, which branch, etc. There is a binary one, but you should use the Python one.
dotfiles: define a repository of dotfiles that you have in a vcs, show what dotfiles you have managed already and allow you to sync or override them, especially handy when on a new system.
On Tuesday 20 September 2011 the yearly Dutch Plone User Day 2011 (Dutch link) was held in De Balie in Amsterdam. This day was organized by various Dutch Plone companies. Next to good conversations during coffee and lunch the day was of course filled with presentations. Here is a short summary of those talks.
Dutch readers are encouraged to read the Dutch summary with links to larger Dutch summaries of the individual talks.
The future of Plone
Maarten Kling from Four Digits talked about "Plone Next", the future of Plone. In which exact version all changes will be available has not been decided yet.
Plone 4.1 is the most recent version. A second alpha release of Plone 4.2 is available with among other changes a better display of search results and new collections that can be managed in an easier way. Next to plans for Plone 4.3 there are more exciting plans for somewhere in the future, whether that is going to be called Plone 5 or something else: a handier interface (Deco) with more freedom for the web master; easier theming of a website; CMS UI: a bar at the top of the page with buttons for editing the page, workflow, adding content, etcetera. The rest of the week we will be working on this with several developers during the Living Statues Sprint.
SalesForce
Wieteke den Uijl from Proteon and Pierre Broekarts from Unit 4 Consist present a case study about the integration of SalesForce in Plone. Unit 4 Consist uses SalesForce as CRM system for internal usage and for marketing, sales, servicedesk, reports. In the old situation there was a lot of manual work, so errors were made. The customer also had no insight in the status of his requests.
So they had wishes: full CMS functionality and automatic input into SalesForce. Plone was chosen for this, with extra functionality for the integration with SalesForce. Customers now know where they can find everything: on this one system.
Books of reference in Plone: publishing without worries
Jan Murre from Pareto presents a case study about books of reference in Plone. KNMP publishes the Informatorium Medicamentorum, a book with thousands of large pages with medical information. We now manage this within Plone. We have built custom contenttypes, are using roles, permissions, workflow, check-out/check-in (for example you are editing the homepage and let others check your changes while the original version is still visible), a WYSIWYG-editor and full-text search. A nice overview of what is possible with Plone in this area.
Lightning talks
Following tradition the official part of the day ended with lightning talks of about five minutes.
- Paul Roeland: from Joomla to Plone. http://schonekleren.nl is currently still an old Joomla site. This will become Plone. We will use transmogrifier to migrate the content.
- Yadi Dragtsma: Free publicity. You have a nice press release but even the small local paper does not publish it. What must you do? A few redactional laws and game tips.
- Kees Hink: Diazo. For http://www.iederz.nl/ we have created a new website in Plone, with Diazo as theming method. Diazo takes pieces from the website (Plone) en places them in the html theme.
- Maurits van Rees (that's me): collective.depositbox. A small python package to store a secret key that makes actions possible, in the same way as resetting your password in Plone.
- Thijs Jonkman: Diazo. For the theming of a site with Diazo we have used the Compass style framework (with Ruby) to create some css stylesheets that work pretty well for most Plone sites, so new sites will be easier to theme.
All in all it was a nice day on which users and developers of Plone in the Netherlands have met.
Armin Ronacher gives the keynote "State of webdev in Python", at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
I am founding member of the Pocoo Team, doing Jinja2, Crossroads, Werkzeug, etc. Python 3 is kind of the elephant in the room that no one is talking about. So Python in 2011 has the big Python 2 versus 3 debate. 'Unladen Swallow' is resting, Python 3.2 was released, the packaging infrastructure is being worked on, including distutils2.
PyPy has become really fast. PyPy is Python written in Python. PyPy trunk is on average 3.7 times faster than the standard CPython, see http://speed.pypy.com. There is only experimental support for the Python C API. Different garbage collection behavior, no reference counting. So a few things will break, depending on what you are using it for. Django, ctypes, pyglet, twisted, all work.
All language development is now happening on Python 3. It adds unicode to the whole stack. The language is cleaned up. It does break backwards compatibility. Most code does not run on both 2 and 3, but a few packages do (lxml, at least at some point). It introduces unicode in exceptions and source compilation as well as identifiers (although I urge you not to try that). Greatly improved I/O API for unicode. Some new language constructs. Implementation was cleaned up a lot.
New constructs: extended iterable unpacking (a, *b = some_iterable), keyword-only arguments, nonlocal, function parameter and return value annotations (use them e.g. for documentation purposes).
print is now a function. Improved syntax for catching and raising exceptions. Ellipsis (...) syntax element.
Some different behaviours. More powerful metaclasses. List comprehensions are closer to generators now. Lesson: don't rely on undocumented 'features'.
Classic classes are gone. Imports are absolute by default.
Python 2.6 and 2.7 make it possible to write code that you can run through the 2to3 program to turn it into Python 3 code. You can use abstract base classes to check for certain implementations (not: is this a dict, but: is this dict-like).
Do you want beauty of code? Use Python 3. Do you want raw speed? Use PyPy.
Numeric libraries work great on Python 3 and benefit from improvements in the language.
Predictions:
- Most people will write their code against 2.7 with the intention of supporting PyPy.
- Libraries that require the C API will become less common.
- We will see libraries that support targeting both Python 2.7 and 3.x.
Now the second part of this talk: Python and the Web.
WSGI has a new specification for Python 3. Some work is done to port implementations to Python 3. It just works; not really an issue anymore.
New developments: Improvements to PyPy support for database adapters. Improvements in template compilation in e.g. Django to take advantage of PyPy's behaviour. Some libraries are being ported over to Python 3.
Python 3 can work. You can start porting libraries over. Issues will only be resolved if you actually try to port. Higher level code is usually easier to port; low level libraries are trickier. Porting is easier if you drop support for 2.6. For porting, see http://bit.ly/python3-now.
WSGI works well in practice on Python 3. Pylons and BFG are now Pyramid, which is a nice introduction into the Zope world. There is less and less framework specific code out there; it is easier to share code.
At the low level, Werkzeug and WebOb may merge at some point; they are much alike.
Frameworks are good. In new frameworks we can explore new paradigms and concepts. It is surprisingly easy to switch frameworks or parts of them. Frameworks themselves are even merging.
I think PyPy will gain more traction in the python web world. It may eventually become more popular than CPython. Supporting Python 3 in PyPy should be easier than Python 2.
Things like 0MQ (Zero MQ) may help to have parts of your code in Python 2 and part in Python 3.
Jobert Abma, ethical hacker at Online24, talks about the ten commandments of security, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
I will discuss ten things you need to think of to get a secure application.
1. Your application is not the only attack vector. There can be weak passwords in other parts of the stack or server. Social engineering can become an issue.
2. Conduct risk assessments to identify risks. Then you start controlling them. You can score a risk on Confidentiality, Integrity, Availability.
3. Only trust your own code. And double check. The platform you are developing on can have security problems.
4. 'Security by design' solves major issues. Application logic is an important part. Centralize validation.
5. Always be aware of technical issues, like CSRF, XSS.
6. Time (mis)management. You don't always get time from your manager to solve security issues, even when you are aware of it.
7. Keep track of design documents and documentation. Is the design secure? Does it still match the current functionality?
8. Process designing is one of the most important processes securing an application. If a checkout process in a web shop is not designed well so that 10,000 euros each day end up on someone else's bank account, that is a problem.
9 Security can clash with usability. 'This email is not in our database' is potentially interesting knowledge for an attacker.
10. Information is power. Encryption on the server side and on the transport layer. If your database gets hacked, does that give the attacker information he can use, like passwords and credit card numbers?
One more thing: handle input as being dangerous. It will save your ass more than once.
Summary: Security is not just a bunch of tricks. It is a process.
Òscar Vilaplana (Paylogic) talks about ØMQ (Zero MQ), at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
0MQ is sockets how it should be. Bindings are available in many, many languages, including Python. Of course messaging looks simple: you send a message and the guy on the other end receives it! Well, it still requires work. But 0MQ is indeed simple. (In python: import zmq). Messages are sent in the background. You have a queue. If the receiver is not there, the message just stays in the queue longer.
Messages are strings and they have a length. With multiparts they can have a sender. You can send through TCP or UDP. You can publish and subscribe. You send messages down a pipeline. As infrastructure you can choose a queue, a forwarder or a streamer.
You can poll from a 0MQ socket or a regular TCP socket or stdin.
Code from this talk: http://oscarvilaplana.cat/zmqtalk.tar.gz
Pieter Noordhuis: Redis in Practice, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
Redis is a key-value store. It can be compared to memcached. But it natively supports strings, lists, sets, sorted sets and hashes. Everything is stored in memory, so that puts a limit on what you can put in it but also makes it very fast. You can also persist it though, unlike memcached. Supports replication, so you can have one master that you write to and say fifty slaves just for reading.
Any blob will do: ascii, utf-8, png. Example:
redis> set str "hello world" OK redis> get str "hello world"
It runs in a single thread: no race conditions or locks; every operation is atomic. This greatly simplifies replication.
Invalidate immediately:
redis> del page:/home (integer) 1
You can have rate limiting:
INCR limit EXPIRE limit 60 # iff INCR == 1
Lists are a natural fit: RPUSH a new job (push it at the right of the list) and LPOP a job from the left of the list. With PubSub you can set up some notifications when jobs are done.
Sets: unordered sets of unique values. Ordered Sets of unique values: ZADD an item to increase its score, ZREM to decrease it. You can use this easily to show currently logged in users, top users for some measurement, etc.
We can do 100,000 gets and sets per second on commodity hardware. With tweaking and better hardware we have heard of 1.5 million per second.
You can have durability through snapshotting: save every N seconds or every N changes.
Getting started: http://redis.io/download, no dependencies, make it and you are done.
Henk Doornbos (Paylogic) talks about making large, untested code bases testable, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
Computer scientist, software engineer, architect, consultant, head architecture department. The Paylogic system is front and back office for payment. Mostly Python (84406 lines of code) and javascript.
There are many abstraction levels: components, classes, features, the code level. Process when doing a support case: staging, testing, bug reports. We use FogBugz as bug tracker. Example: downloading a 'scanware' file was broken. Problem solved now, but can we check automatically that it has indeed been fixed and is not broken later again. Maybe we can execute the bug report as a test? This can be done when you are rigid enough in specifying the bug report ('Given When Then' style: starting state, steps to reproduce, expected behavior). You can write a small language that specifies what should be done when you encounter specific text in a bug report, translating that to Selenium tests (visit this page, click there, check that text is on the resulting page). We use the robot framework.
You could do similar things with record and playback. But often the bug report is a missing requirement. We only want to add features that deliver value. Behavior driven development style: to have some fun (goal, value), as a customer (who, actor), I want to buy a ticket (reason). You can write that as:
- Given I am buying tickets
- When I choose the amount
- Then I see the total cost
So in one smooth, agile process you create these products: requirements document, acceptence tests, working code that delivers value, bug reports.
Gideon de Kok talks about Mobile Architectures, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
Yes, you should treat mobile clients differently. Mobile applications should still work reasonably even when not connected, if possible. You want to get things done quickly. More push and pull. Best practices are still mostly the same. Difference: your server should work harder than the client: no big calculations in javascript. Don't let the client pull but let the server push; this saves battery life.
Improve connectivity: do lots of caching. Store them locally in the browser/app. Refresh your data periodically, also on user request. Properly expire and delete data. Combine requests so you have less problems with latency and connectivity. If connection fails, just wait. Queue requests and send them when connectivity is there again, instead of checking every few seconds. Restrict connectivity: if you don't use it, don't send it. Don't preload things the client may possibly need.
Make your payload small, one simple trick is of course using gzip. Adapt your payload: send smaller images or send only alternative text. No more direct database access: create an API to talk to the database, including logic for failing connectivity. Never trust API input. Client side validation is good for connectivity and performance, but you definitely need to check on the server too.
Security: encrypt your data streams and store secret stuff well. And do not save too secret stuff on the client: you don't want thieves stealing secrets from your phone by reverse engineering.
A mobile-specific API probably has more push and less pull calls.
Don't overengineer your SSL connections. It takes too long to decrypt it on a mobile device if you send too much encrypted data. Safety versus speed.
Native applications are generally speedier and better in the pushing and pulling, relying less on good connectivity than a mobile web site. When you build native apps as e.g. a bank, you suddenly become a software creator, instead of just needing to keep a website running. A web app based on html5 and css3 it will mostly work on all devices; with a mobile specific app, you have lots more devices that you would need to check for compatibility. There are frameworks for this, but the resulting apps will be less effective than an app created specifically for one device.
Luit van Drongelen talks about Lightweight Python deployment servers, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
I code python for fun (and hopefully eventually for profit). What is wrong with Apache? Well, nginx is much faster. You can double the responses and have much less memory usage. Apache has higher I/O load, e.g. for loading .htaccess file. nginx can natively connect to uWSGI: fast, light weight version of modwsgi. The protocol is uwsgi (so all lowercase). This is run separately from the web server. Tested on many operating system; not Windows, sadly.
So why use uWSGI instead of e.g. modwsgi. It's fast with a lower memory foot print. Can handle multiple interpreteer versions in multiple virtualenvs. Supports old and new WSGI standard. It can kill misbehaving worker threads; helps if you are coding those wrongly yourself. It can also handle long-running tasks. Configuration can be in ini files, json, environment variables, command line options, etcetera. Built-in message-passing system. Embedded (evented/async) HTTP server. It even has a clustering feature (beta at the moment). [Live demo of that feature, which worked yesterday.]
Some other WSGI servers show great performance too. For me uWSGI performs great and has good features.
Comment from room: look at this WSGI performance comparison.
See the WSGI slides.
Rix Groeneboom (Parasoft) talks about Mijn Overheid: Performance testing in practice, at the PyGrunn conference in Groningen, The Netherlands. Organized by Paylogic and Goldmund Wyldebeast & Wunderliebe.
('Mijn Overheid' is Dutch for 'My Government'.) I have a new hobby: collecting screen shots. One is a screen shot of 23 March 2010: disruption at DigiD, central login application of the government, for example for submitting your tax form. Another one: Servers of Cito offline during exam. Yet another one: at the DUO website (government agency that gives loans to students) you could see data from other students. And: changes in a test environment were available on other production systems, so people suddenly discovered they were married or had children.
At the Mijn Overheid website you can login with your DigiD and see all kinds of info about yourself, like home address, home ownership, license plates, speeding tickets. If you agree, the government can send you messages in the system instead of via paper mail. The government wanted a way on this website to know for sure that you have read it. The system is called the 'berichtenbox', message box.
We wanted to be able to simulate lots of civilians ('burgers') to login and use the web site and see what that does with the load on the servers; what is the 'temperature' of the system? Systems in isolation were tested, but there were also dependencies and those make it trickier.
With the government we specified performance requirements. We profiled the server implementation (a LAMP setup). Some search queries were very slow as they needed info from several servers; searching for 'gemeente' (community, city) would give almost every item back as search result. We profiled the combination of the MO (Mijn Overheid) website with the GEB (database with info on civilians).
We had only 2 DigiD accounts available to test the load on the system. That is not quite enough. So we faked/stubbed the DigiD server. This way we could let the test MO website talk with our fake DigiD server with lots more load, making the testing also simpler.
We created a platform for SOA and chain testing. Written in Java, Jython and Eclipse. Agile and continuous integration with a command line and web interface. It could run on Windows, Linux, Solaris, with a Linux VMWare image available. External integration with version control, defect tracking and testdata generation.
For the load testing we used Python and MySQL. We implemented intelligence, like simple 'wait' steps, generating valid BSN (social security) numbers. Every request was logged in MySQL; this database also had a shadow administration of user accounts. Extra checks to see if someone had been able to see information for a different account.
The system was very fast on low load. But at 1000 users at the same time, requests could take 20 seconds. That would be quite an extreme load, but good for testing. We found that the database query took 90 percent of the time. Also loading the PHP modules at some point took 8 seconds. A little smarter programming fixed that (being smarter so you don't always need require_once in the interpreter). Some optimizations in NFS, Apache and the OS.
We talked about process improvements. The load testing sometimes could not be done on the acceptance server because that was being used to test functionality. We figured out where the actual bottle necks were, which could be on a different system: it's nice if the load on our server is low, but if the user is still waiting for reaction from a different system it does not buy him much.
Summary: We set a performance norm. We found out the weakest link in the various system. Result was a large number of improvements.
On 20 May 2011 Paylogic and Goldmund Wyldebeast & Wunderliebe hosted the second PyGrunn conference in Groningen, The Netherlands. The conference was about the Python programming language and related technologies. There were two and sometimes three tracks of talks. Here are links to the summaries I made. See my brother's weblog for his (partially overlapping) summaries.
- Mijn Overheid: Performance testing in practice, by Rix Groeneboom
- Lightweight Python deployment servers, by Luit van Drongelen
- Mobile Architectures, by Gideon de Kok
- Making large, untested code bases testable, by Henk Doornbos
- Redis in Practice, by Pieter Noordhuis
- ØMQ (Zero MQ), by Òscar Vilaplana
- The ten commandments of security, by Jobert Abma
- State of webdev in Python, keynote by Armin Ronacher
Peter Hilton: Welcome to Lunatech
Sorry, I have not written any python myself. ;-) Technically focused company. We do almost everything in Java. We use an open source stack. So not very different in mindset, we just don't do a lot with python. Focused on back end stuff. Commercial IT services. Ten years ago we used Perl. I don't know why we switched to Java as I was not there at the time. I also don't know why we stay with Java. Actually, Scala is a more likely successor for us than python. We make money, which allows us to put this beer on the table. We hire hard-code geek programmers, creative rocket scientists, people who get things done, tech wizards with ambition, obsessive-compulsive types, managers in suits (not!!!). Python paradox: potentially you have more good candidates than in Java as there are so many bad Java programmers. We have a coding question for job interviews. We had to make it simpler or too many people would fail. It was not a question of: pass this test and you are hired, but pass this test and we can continue with the job interview.
Kit Blake (Infrae): VisitorVu: Real-Time Visitor Tracking for Your Website
http://brusselnieuws.be. Three different news sources coming in. They wanted to know real-time what was going on on the website. Direct hits on the web site, searches, external links, internal links. Popularity. Tag cloud. We are considering offering this as SaaS; we are probing if there is interest in the market. So maybe you or your clients? See http://visitor.vu/ and the code.
Steve Alexander: Interactive code previews
I ran a large software team at Canonical. Mostly I was hiring Python programmers. I was always looking for existing code that interviewees had already written and that was out there being used. Then I quit my job and took acting classes, did coaching, etcetera. How do people communicate, that's what I am busy with.
Seizing the task. Fix a bug, optimize, write a blog post, all are tasks. Rubber duck programming: you buy a rubber duck and explain him the process of what you are going to do. By talking and hearing yourself back (or maybe writing and reading it) you focus and it becomes easier to see if it still makes sense. Geir Baekholt at Jarn: "When doing a task, we talk it through with another person." Are you a good listener. Presume a lot: they have expertise (else why are they working here), they can find things out for themselves, are creative. Destroy ambivalence: not maybe/possibly/whatever, but yes or no. If an issue is in a bug tracker for too long it sucks the life blood out of a company, product, community. Motivational interviewing: don't try to make someone enthousiastic or offer 'helpful' solutions that the idiot other has probably not thought about, but ask him about his own motivation and how he could make himself more motivated. I started as a very technical manager, which made people look up to me too much as I could maybe do their job better. If you hire someone, believe in his skills and trust him; try to find smarter people than you.
You see a lot of small software companies in The Netherlands; that is cool. Every company seems to have its own culture. It is important to attract good people: I want to work with people I want to work with.
![Steve Alexander presenting at PUN [Photo by Jasper Spaans]](images/steve-alexander-pun-19-may.jpg)
Steve Alexander presenting at PUN [Photo by Jasper Spaans]
Matthijs Kadijk: Some nice things you can do with google data APIs
I am an independent software developer. Just moved to the Creative Factory in the Maassilo in Rotterdam. Most of the Google services have AP Is that you can talk to within your program. RESTFul web services, an xml version based on ATOM feeds, json available too using alt=json parameter, OAuth authentication. Some use cases: schedule an SMS (text) message, add an appointment to a calendar, import data from Office docs without having Office, perform OCR, create a PDF invoice. Google code has a python client lib gdata, available via PyPI: http://pypi.python.org/pypi/gdata The python implementation does lag behind the other implementations a bit, so maybe some of us can help? The services can respond slowly, so watch your timeouts. Read the source code of the library to find non-documented features. Let the user do a manual login first, otherwise you can get a captcha. Use stored access tokens to prevent blocking after too many login requests. And you may want to consult an expert when you run into problems.
Nicolas (Lunatech): Play! framework and Python, what's the deal?
The Play framework makes it easier to build Web applications with Java and Scala. Could be Jython with the right modules. Lunatech has contributed to this. It is inspired by Django and Ruby on Rails. The scripting aspect is done in Python. We try to be language agnostic. You don't need to restart the server when you change your Java code; this is done on the fly. There is a community around it that develops extra modules. See http://www.playframework.org/
Sylvain Viollon (Infrae): Demo of infrae.testbrowser
With infrae.testbrowser you can test web applications that run on WSGI. You can ask the test browser to look for content with xpath expressions or css selection. So you can more easily make sure you don't get an error when there are two inputs with the same name. It can also talk to Selenium and execute the tests in a real browser.
Jan-Jaap Driessen: Hackathon
We did a hackathon/sprint in Rotterdam two weeks ago. About ten people showed up. Was a good day. I can set up a new meeting but others can do that too. I am planning one in a week about deployment. Keep an eye on the mailing list of PUN. Let's learn from one another and exchange ideas and code together.
![Beer and pizza at the PUN [Photo by Jasper Spaans]](images/beer-pizza-pun-19-may.jpg)
Beer and pizza at the PUN [Photo by Jasper Spaans]
My brother Reinout (who was the main organiser) also has summaries.
Christo Butcher (Fox IT): Trac with python
Sorry, I came in late and missed the first part of this talk and was eating during the second part. :-)
Maurits van Rees (Zest Software): Theming with xdv, Paste and fanstatic
Since a few days I use xdv, Paste and fanstatic to put a new theme around my old Plone 2.5 website. The code is available on github and can be instructive as an example of these technologies: https://github.com/mauritsvanrees/maurits-site-xdv
See the slides.
Jan-Wijbrand Kolman (The Health Agency): Publish your changes
I do a lot of releases, especially for Grok and the Zope Toolkit. So I look a lot at the Python Package Index (PyPI) to see which changes have been made in packages.
Quote: "Zope Community and Friends: I love you so much for putting changelogs in your PyPI info." (jshell)
Compare e.g. Sphinx and grokcore.component on PyPI. Sphinx hardly says anything on its PyPI page. grokcore.component is much smaller but publishes a lot more info right there on PyPI, especially its changelog. Add that to your packages and give them a meaningful package description. It is probably the first page someone sees for most packages, so you had better leave a good first impression.
Jasper Spaans (Fox IT): pypy rocks, even in the real world
(Note: pypy is not PyPI.) Pypy puts a python in your pythons. It is a python interpreter implemented in RPython, restricted python. You can do memory management also in RPython, so you don't need to handle, understand and write C code if you need your own specific memory management. It should be a drop-in replacement for normal python (CPython) in most cases. It offers a possibility for a JIT optimizer, which can make your code faster.
We tried pypy a few weeks ago. We encountered some SyntaxErrors, as version 1.4.1 only supports python 2.5). Class decorators are not supported yet (so we decorated manually). The with-statement needs to be imported from __future__. Memory usage exploded (fixed by changing to the hybrid garbage collection). SQLAlchemy was too hairy; for that part we put in a bit of standard python to handle the SQLAlchemy integration, together with protobuf.
Several of our performance tests showed improvements of 7 to 13 percent when using pypy. The memory usage was 20 percent higher though.
Reinout van Rees (Nelen & Schuurmans): Geographic information websites for water management
Our customers are mostly government agencies working in water management. They have a lot of information about water. We make that info available in a format that is easier for them, on a website. Customers can click through maps in our application to get info. What technology do we use? Lots of python!
- Mapnik: uses the rendering engine from Openstreetmap. You have points, lines and grids. It understands the WMS standard for web mapping. It gives a .png file as output.
- Gdal: used behind the scenes mostly, for grids.
- Matplotlib: a graph library. It can do everything. Others are easier, but wit limitations. Several customes have too many peculiar wishes that are difficult to do with other packages, if it is possible at all. (Remark from the audience: use protovis.)
- Pyproj. You have various measurements for where on the globe you are: rijksdriehoek (in the Netherlands), degrees, Google mercator. Pyproj translates between these.
- On the client side we use jquery, openlayers, and the blueprint css framework.
We have a layered software structure:
- core: splitting everything up.
- lizard-ui: defines the html page structure.
- lizard-map: this is the big python/django application that we have, the core of the map. Map visualisation; basic graph and search handling; basic popups; plugin mechanism for adapters (using extra attributes in html5) with which you can search, get a layer and show the corresponding html in a popup.
Organisation:
- We have collective code ownership. Everyone can touch all the code. For most of the code everyone has a basic understanding of what is going on. We want code that conforms to the pep8 and pyflakes standards. For automatically running the tests we use Jenkins (formerly known as Hudson). We borrow each others brains: sit with each other to check ideas that you have for sanity.
- Documentation is generated by Sphinx.
Our clients are mostly governmental and they like it when they use open source, so we have open sourced our code, also because it uses a lot of open source itself. This also attracts me to the company.
See http://doc.lizardsystem.nl/ and http://reinout.vanrees.org/weblog/
We want you! Business is booming! See http://www.nelen-schuurmans.nl/
Coen Nengerman (Nelen & Schuurmans): ArcGIS
I studied hydrology; I am more a python user, not a developer. ArcGIS is a professional tool to manage your geographical information. It makes lots of info and tools available in a nice UI. Catalog, tools, python scripts using e.g. scipy and numpy, including an integrated python command line. It can find maps online as well, like buienradar or maps of the flooding in Pakistan.
Jan-Jaap Driessen (The Health Agency): fanstatic
Fanstatic is a python package for resource management: css and javascript files. I am using zope and grok. Getting resources from zope can be slow, because lots of checks are done, for example authentication and authorization, which is not always needed. Usually you include too much resources on all pages, because you do not know for sure which of them you need for a specific page. Fanstatic fixes this for you. The fanstatic middleware will figure out which resources you really need and serve them to you.
You could download a jquery package from somewhere and put it in your own code, but we want to create python packages for this so you can easy_install them (js.jquery, js.yui). There is no real management of dependencies in the javascript world currently like there is for python, so for now we create python packages that define these dependencies.
Fanstatic minimizes http requests by doing rollup, bundling, using a cdn (content delivery network that hosts common javascript resources) if available, and versioning a resource so it can be cached for ten years in a browser without getting stale: when you change the resource a version with a new id is requested in the html. It takes hints from Steve Souders at http://developer.yahoo.com/performance/rules.html
On the roadmap (some already on branches): bundling, cdn, compile for minification (google closure compiler), lazy loading (google loader, requirejs), commonjs.
Nico de Groot (Tilburg School of Theology): web2py
web2py is easy to learn, stable, secure, light weight, faster than zope ;-), can use different kinds of databases, can run inside Google Apps, some mercurial integration.
At the top you have applications consisting of plugins. In the middle there is gluon (old name of web2py) and general libraries like simplejson, and at the bottom the rocket server.
Jean-Paul Ladage (Zest Software): Prettig Personeel
Prettig Personeel is an online human resource management system created by Zest Software using Plone. You can generate contracts, get reminders that contracts are ending or for birthdays or for people who have been ill for a long time.
When you view an employee in this web application there are lots of boxes with info in which you can view, add and edit contracts, etc. We developed jquery.pyproxy to handle this. With that, you can use jquery within python (works in Plone and Django). You can use all manipulation and effects of the jQuery API. The result is returned as an xml response for an AJAX call so it gets processed on the client side in the browser.
It's on pypi and github:
See these pages for info on why you may want this: http://plone.org/products/plone/security/advisories/cve-2011-0720 and http://plone.org/documentation/kb/disable-logins-for-a-plone-site
If you want to change lots of nginx config files to temporarily switch off login (authentication) and cookies, you can use this bash script at your own risk:
#! /bin/bash
# Note: /bin/sh would be better, but at least when that points to
# /bin/dash it complains about some of my usage of 'test'.
cat <<EOF
This script looks for nginx files in the current directory. Actually,
it looks for files having at least one occurrence of 'proxy_pass'.
After each line with 'proxy_pass' it inserts two new lines:
proxy_set_header Cookie null;
proxy_set_header Authorization null;
See these pages for info on why you may want this:
http://plone.org/products/plone/security/advisories/cve-2011-0720
http://plone.org/documentation/kb/disable-logins-for-a-plone-site
As a safety measure it copies the original filename.extension to
filename.extension.ori.
NO WARRANTY AT ALL. ASSUME THIS DELETES ALL YOUR FILES!
COPY ALL FILES IN THE CURRENT DIRECTORY TO A BACKUP LOCATION,
PREFERABLY ON A DIFFERENT CONTINENT.
In other words: USE THIS AT YOUR OWN RISK.
Maurits van Rees
EOF
DRYRUN='yes'
if test "x$1" == "xyes"; then
echo "This is NOT a dry run."
DRYRUN=
else
echo "This is a dry run."
echo "Run with '$0 yes' to really change the files."
fi
for CONFFILE in *; do
if test $(grep -c 'proxy_pass' $CONFFILE) -eq 0; then
echo "No match: $CONFFILE"
else
if test $DRYRUN; then
echo "Would change $CONFFILE and keep backup in $CONFFILE.ori"
else
if test -e $CONFFILE.ori; then
echo "NOT changing $CONFFILE: $CONFFILE.ori already exists."
else
echo "Changing $CONFFILE and keeping backup in $CONFFILE.ori"
# 'cp -n' means: do not overwrite an existing file
cp -n $CONFFILE $CONFFILE.ori
sed 's/proxy_pass.*$/&\
proxy_set_header Cookie null;\
proxy_set_header Authorization null;/' $CONFFILE.ori > $CONFFILE
fi
fi
fi
done
echo "Do not forget to reload or restart nginx after changes."