Plone
This is here to serve as contents for the atom/rss feed for Plone, also read by planet.plone.org.
Oleg Pidsadnyi - Behaviour driven design with PyTest
Oleg Pidsadnyi talks about behaviour driven design with PyTest, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
Code of pytest-bdd: https://github.com/olegpidsadnyi/pytest-bdd
I will talk about behaviour driven development in Python. What is it? You define a scenario with strict language, like given this and that, expect this. It is readable for both programmers and business logic persons.
There are several ways to do this. You can use Lettuce or Freshen, plus Splinter. But imperative coding style does not work well here, I think. Given I have two books. How do you do that? context.books = [...]? context.book1, context.book2? pytest-bdd has the concept of expecting and returning values, with @pytest.fixture:
@pytest.fixture def author(): return Author() @pytest.fixture def book(author): return Book(author=author) @given('I have two books') def article(author): return [book(author=author), book(author=author)] @given('I have an article') def article(author): return create_test_article(author=author)
This is much more explicit.
It can do browser tests. You can use a normal browser, like Firefox, or you can set it up headless with phantomjs.
pytest-bdd is inspired by the Robot Framework, but we had some different requirements.
Douwe van der Meij and Brandon Tilstra - MVC revisited with Diazo
Douwe van der Meij and Brandon Tilstra talk about MVC revisited with Diazo, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
We work for Goldmund, Wyldebeast and Wunderliebe, the sponsors of this conference with the longest name.
We use Diazo for theming. Your application creates an html page. Your designer creates a standard html template including css. With Diazo you merge the two. It separates the content creation from the styling.
The technique is plain old XSL transformations. Nothing new.
Why should you use it? Designers and developers look at a certain project in a different way. The designer sees the beautiful outside of the car and the developer sees the gritty details under the hood. Developers usually want to stick to developing features in a minimal design, just some standard html preferably without any css.
If you look at the Model-View-Controller paradigm with Diazo in mind, it makes the View part easier: the developer handles the application part and the designer handles the styling part. The designer does not need to know Django or Plone templates.
Brandon is busy making Diazo available for other applications than just Plone. For Plone a tool is available: plone.app.theming. Since Plone 4.3 you can edit the theme inside Plone. A designer can do that, with a WYSIWYG editor in Plone and a developer can tweak the code with a code editor in Plone.
Brandon is working on thememapper, a standalone theme editor, written in Python. thememapper.core is the tool itself. thememapper.diazo is a diazo server to be used with thememapper.core.
Question: Aren't you trying to solve an organizational problem? Shouldn't the designer and developer be talking to each other?
Answer: We often get a design in Photoshop, give it to a third party front-end party that creates html and css from it, and we as developers create the Diazo rules.
Alessandro Molina - High Performance Web Applications with Python and TurboGears
Alessandro Molina talks about High Performance Web Applications with Python and TurboGears, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
I am @__amol__ on Twitter. I am a member of the TurboGears team. I will talk about general rules which apply to any framework, some quick wins for TurboGears, some real cases, my personal experiences and preferences, feel free to disagree.
People seem obsessed with the raw speed of web servers. But you are not going to serve a "Hello world" page. My personal stack has nginx plus mod_wsgi or nginx plus Circus-Chaussette-gevent.
Try to avoid making everything an asynchronous (AJAX) request. Browsers have limited concurrency. HTPP has overhead. You will actually slow things down if you have too much of them. Your page may start fast but complete slow. Learn your framework and how you can use it the best way for your use case.
TurboGears is a framework for rapid development, encouraging flexibility. It was created in 2005, with 2.0 a major rewrite in 2009 to embrace the WSGI standard. It is based on object dispatching. You can use regular expressions for url matching, but they can get messy, so write them only when you must. By default an XML template engine with error detection. Declarative models with a transactional unit of work. It has built-in validation, authentication, authorization, caching, sessions, migration, etcetera.
Features versus speed: TurboGears is a full-stack framework. That makes it quite slow by default. You can switch things off that you do not need. The team invested effort to constantly speed it up since the 2.1 release. Keeping all the features around has its price, obviously. To cope with this, a minimal mode got introduced, that switches several things off that have the biggest influence on performance. You go from about 900 to 2100 requests per second.
Avoid serving static files in your framework. Let some other part of your stack handle this. This can take a lot of load from your application server.
Use caching. Caching means preorganizing your data the way you are going to use it. For example with a NoSQL database you can load the comments directly when accessing the page, so you don't need to load them separately. Frameworks usually provide various types of caching. Get to know them and use them.
Use HTML5 and Javascript. Invalidating your whole cache just to add a message "Welcome back, mister X" is not a good idea. Cache the result and use Javascript to do minor changes. If you are using varnish, nginx or any other frontend cache, consider using Javascript plus localstorage instead of cookies for trivial customizations, because cookies disrupt the cache.
Cache the result of rendering a template, with a cache key.
Entity caching: cache parts of your page, for example the html of one comment or notification.
Proactively update the page in your cache: when you edit the page, update the cache before your visitors ask for it.
If you are struggling too much with improving performance, you are probably doing something your application is not meant to do. Also, football fans are really eager for updates.
Offload work. Update the core cache to provide the author with an immediate feedback. Let some other process, program or thread handle the related background changes. For example, use something like Celery.
New Relic App Speed Index reports an average of 5.0 seconds of response time for accepted experience: http://newrelic.com/ASI If response time is less than 200 milliseconds, this is seen as 'right now'.
Sorry, there is no silver bullet for speeding up your application.
Kenneth Reitz - Python for humans
Kenneth Reitz talks about Python for humans, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
Follow me on twitter: @kennethreitz_. I work at Heroku. I am a member of the Python software foundation. I created the requests library that makes is easier for humans to retrieve a web page in Python.
I want everything I build to be open source. I build for open source, pretend it is open source even when it will never be released to the public because it is too client specific. Add documentation, please, also to your internal tools.
Open your Python prompt and see the Zen of Python:
>>> import this
Beautiful is better than ugly: you don't need lots of curly braces in Python. Explicit is better than implicit.
Most importantly for today: there should be one - and preferably only one - obvious way to do it.
Welcome to paradise. You have found Python, a language with a beautiful zen philosophy and all is going to be fine. Lies!
Look at some Ruby code to download the contents of an https web page. Perhaps a bit too many lines, but it is pretty straight forward. Now you switch to Python. Which library? urllib, urllib2, something else? Okay, you use urllib2 and then you have to add some password manager and lots more lines that are unclear. You will leave and never come back.
This is a serious problem. I think HTTP should be as simple as a print statement. It is used so often! Our world is connected over HTTP right now so we need this to be simpler.
Python needs more Pragmatic Packages. Deal with things sensibly and realistically, in a way that is based on practical rather than theoretical considerations. Make it practical for humans, for the common case.
The requests library is HTTP for humans. A small set of methods with consistent parameters. Do this for more modules! Fit the ninety percent use case. Features, efficiency, performance, etcetera are important, but you should ignore them for such a practical package. Go for the common case. Write the README the way you think it is supposed to work.
"Cool story, bro, but why should I do with this?" It's worth your time and everyone else's time as a developer.
Some things that could be improved:
- File and system operations. We have the modules, sys, shutils, os, os.path, io. Which should you use for a task? It is really difficult to run external commands and this blocks dev+ops folks from using Python.
- For installing Python there are various ways. Use the Python that came with the system or compile your own? Python 2 or 3?
- XML hell. etree annoys people. lxml is awesome, but can be difficult to install.
- Packaging and dependencies. Pip or easy_install? How about an easy_uninstall? Distribute or setuptools? [Holger Krekel will talk about this topic later today in the keynote.]
- Dates, datetimes. Which module: datetime, date, time, calendar, dateutil, version 1.5? Timezones, they are ridiculous.
- Unicode. We'll just skip that one.
- Testing. Lots of different ways to create tests. Doctests and unit tests. Various test runner libraries, like nose and tox.
- Installing dependencies. Python-mysql, if you remember the exact name. (My solution: just use Postgres.) Python Imaging Library needs several system packages before you can install it. mod_wsgi: if you install this, which Python version are you using, the one from the system?
Hitchhiker's guide to Python: http://python-guide.org The goal of this project is to write down the tribal logic, documenting best practices. A guide book for newcomers. A reference manual for seasoned pros. It tries to document the one - and preferably only one - obvious way to do it. Please contribute documentation here. This lets of practice what we preach.
Lets fix our APIs and improve our documentation.
requests might get into core at some point.
requests started as a wrapper around urllib2. It was a nightmare. It now uses http to handle the lower level parts.
If I would have said 'yes' to all feature proposals for requests, it would have been too much. People can write a library around requests. Saying 'no' made it possible to create and maintain a good architecture.
Dutch Python Web Meetup 21 March 2013
Hosted in the Travelbird office, Amsterdam
Remco Wendt starts this meeting off with some bad news.
Malcolm Tredinnick passed away. He was a really major contributor to Django. He will be a big loss to the community.
A UK company is claiming a trademark over Python. Ah: it has been resolved.
The Django sprint in Utrecht was a big success. It has just been decided at the general Dutch Django meeting to repeat this next year: 22 and 23 February 2014.
Orne Brocaar - Job-Runner
Job-Runner is a product we built inside Spil. I am a freelance Python/Django web application developer. I am currently working for Spil Games, a gaming company in Hilversum, about 70 people working there, about 50 gaming portals worldwide.
I am working within the data warehouse team. We gather data from internal and external sources, like analytics data, game related data. We convert this into a Data Vault and Data Mart model, where we can analyze it so we can understand our users. We create reports to inform the business. The data is imported into the staging area (database), then the data warehouse, then we go to reporting, analysis, recommendations. These are all jobs that need to be performed in order.
Before Job-Runner this was all scheduled by cron. If one job failed, the follow-up jobs would fail too. There was no good overview of running, completed and failed jobs, hard to quickly see what (if anything) went wrong. Cron could start a new job if the previous long-running job was still running. Not good.
Job-Runner is a centralized dashboard, an admin interface, a broadcaster for jobs, a RESTful interface. Then there are decentralized workers (job-runner-worker), and a centralized WebSocket server (job-runner-ws-service). Those two communicate with Job-Runner.
The dashboard is HTTP, the queue broadcaster is ZeroMQ.
A project has a job template, for example a python snippet with some variables that are filled in. A job is created from this template. The job is executed in a worker pool. The job may run multiple times, for example once every day. You can also start it manually from the dashboard.
Job-Runner is a Django project, with TastyPie for the RESTful interface. There is a long running management command (the publisher), called broadcast_queue. The worker subscribes to this publisher. It publishes events to ZeroMQ. gevent handles the event queue. The WS Server is a simple proxy between ZeroMQ and the WebSocket server.
We started in December last year. We looked around for other solutions, but they did not fit our needs. The authentication system of Django worked nicely for us. You can setup notification e-mail addresses for a job or pool. Via the admin interface you can schedule jobs at a specific time. If a previous job run has not finished yet, the new job instance will not yet be started.
We have used this for hundreds or even tens of thousands of jobs. We only have one normal cronjob left.
The documentation is at http://job-runner.readthedocs.org
Personal website: http://www.brocaar.com
Jan-Jaap Driessen - Sentry
I work at Mind District. We set up Sentry in our web stack, because we don't want to hunt around for stuff in lots of logs on several servers.
Tip: use UDP for sending the logs to the central server.
We have used this in Python, but are now also using it in javascript, as simple as this:
MD.logging.warn('bar');
Inspired by Strophe. We have a few lines of code for that. Ask me. We also use log4javascript, which is 128 kB and that is minified. In development you want to use the console, but in deployment you want to use Sentry. The code uses Raven to capture the logged message and send the message to the Sentry server.
The WSGI integration is in Zope3, let me know if you are interested.
Diederik van der Boor - 10 things to take care of when Open Sourcing your package
For example django-fluent-pages, developed by me.
- License: do not use the GPL, but the BSD or the Apache license, otherwise you cut your project off from other Python community modules. [For the record: I do not agree with this. Maurits]
- Please provide a README with a reason for using your project. Maybe a screenshot.
- A changelog please. I don't want to look at github commits to check if my bug got fixed in a specific version.
- Documentation. For example on http://readthedocs.org. Look into sphinx.ext.autodoc for automatically generating documentation from your code. Also graphviz and intersphinx.
- Add tests. A runtests.py that is enough to run your tests without needing to setup anything is a good idea. Look into Travis for automatically running tests on a server. tox is good for running tests locally.
- Use the communication channels. Keep people updated. Have a website, with documentation.
- Please provide an example project, so people can easily see how it works.
- Create a good setup.py.
See my github account: https://github.com/edoburu
I count eight things, not ten, so I apparently missed two. Well, two points from the audience follow.
Audience:
- If you start losing interest in your project, look for someone who can take over, preferably someone who has already created some pull requests. Also give people commit access to your repository if they have made some good pull requests.
- Upload your distributions to PyPI, not your own small server. And leave all versions online, otherwise some people will be very annoyed as they are using it on their live site.
Eric - Two Scoops of Django
Daniel Greenfeld and Audrey Roy are writing a book on best practices for Django development. It is in beta, but you can already buy it. It is very good, so please do. The beta is 14 dollars and then you get the final version for free.
Get it at https://django.2scoops.org
Jan Murre - Webapp2
I am active with Python, Zope, Plone, Django, since 1999. I am currently working for a company in Utrecht and before that Pareto.
Webapp2 is a lightweight Python web application framework. It is compatible with webapp, its predecessor, which is the original Google App Engine framework. Webapp2 has several improvements, for example better URI routing and WebOb. See the webapp2_extras package for more stuff. Google has adopted it since GAE SDK 1.6.0.
It has WSGI integration. You can choose your templating language: Jinja2, Mako, etcetera. Forms: wtform, formish, etcetera. Persistence: for AppEngine there is BigTable. Stand-alone you can again use whatever you like: SQLAlchemy, some noSQL, etc. Use a WSGI container, like uWSGI, mod_wsgi, paste. So you pick and choose your own stuff on top of the micro framework.
Google App Engine is Platform As A Service (PAAS). You upload your application there. Comparable in that respect with Heroku. Easy to build, easy to maintain, scales and load-balances automatically, you pay for what you are using. There are several quotas and limits. For some applications we only have to pay the minimum of two dollars a week for the professional version. There is also a free version, with low limits for bandwidth, memory. Languages supported are Python, Java, Go (experimental). BigTable is a noSQL storage, with queries, sorting, transaction, caching built in. URL fetching. E-mailing. There are task queues (think: Celery). Scheduled tasks (think: cron).
You start the application with a simple, plain text yaml configuration. Of course I use Buildout to setup my local development environment, where you have some local storage. The recipe to use is rod.recipe.appengine.
There are other Python (micro) frameworks that you can look at, for example Flask, Pyramid. You can use those on the Google App Engine too, but it takes a bit more setup.
Conclusions for Webapp2: small, but nice. Quite some batteries included. Good routing engine. Plays nice with Google App Engine.
Together with someone else I created http://mijnvoetbaltrainer.nl that uses this, in combination with Amazon for hosting the video material.
More info on webapp2: http://webapp-improved.appspot.com
Pepijn Vos - Pygments, ctags
Pygments: code colorizer. Ctags: allows you to jump to a definition of some code. You can combine that. I did that in spelunking. You can use it to browse github repositories and click around in it. It downloads the repository and runs Pygments and ctags on it and makes it available.
Code: https://github.com/pepijndevos/spelunking
Demo server: http://raspi.pepijndevos.nl
Jan-Jaap Driessen - video conferencing
We have built some cool stuff for health care professionals who visit other people. It can be handy to do this via video communication. We did not base this on Skype, but on Google video conferencing. http://videmo.minddistrict.com There was a SOAP api that we had to interact with. If you have to do that, you should use suds. Using bosh, jabber, xmpp on the server. You can use it simply with an Android phone, no need to buy special hardware.