Weblog
Gil Forcada Codinachs: Beyond QA
Talk by Gil Forcada Codinachs at the Plone Conference 2015 in Bucharest.
Look at these Python packages: pep8, pyflakes, flake8, bobtemplates.plone, plone.recipe.codeanalysis. Sysadmins like to be lazy, or should be lazy: he does not want to get a call at midnight to get back to work putting out a fire, because everything is running smoothly, and anyone can do it because everything is setup correctly.
From one colleague I used to need to check all his work, even after explaining, explaining, explaining. There might be tests, maybe not, maybe failing, non standard code, etcetera.
Instead of spending time on real reviewing, it turned into a fight about how to do things, which style to use. So I stopped complaining. I did it differently: let the machines work for you. Let continuous integration tell you all the things.
Are tests passing? Pull requests on Plone are passed through our Jenkins test server.
Does code adhere to a style guide? Is it automatically checked, also by Jenkins or Travis?
QA is about asking why? Why did you make the changes this way? Why did you create this new buildout part? Step back from the line-to-line review and get the full overview. Maybe the change should be in a new package, or the change should not be done at all.
Side note: NASA's take on software development. See http://www.fastcompany.com/28121/they-write-right-stuff
End user or developer documentation. Maybe you create a new option for a buildout recipe, but nobody knows about it unless you document it. Maybe you write everything in English and not everyone can read it. Does it run in all browsers, did you test that? Maybe there is a broken layout, because you forgot to add an upgrade step. Is a dependency missing?
Side note: for checking layouts, see huxley, screenster, PhantomCSS.
So how do we fix it?
Have a style guide and conventions, some documentation on how your team wants the code to look like, what best practices you should adhere to. The best style guide is a coded guide, an automated check, especially as a commit hook. Use the plone.recipe.codeanalysis[recommended] recipe. Use flake8, not only in this recipe, but look at it yourself, integrate it in your editor, look at all the plugins that have been created for it. Maybe create one of your own, it is not hard.
Use a CI (continuous integration) system, it saves time. It is a newcomer's best friend, or even the seasoned veteran's best friend.
Fill your toolbox with zest.releaser, z3c.dependencychecker, lots more. Look for tools, share tools and experiences that help you adhere to the style guides and best practices. Improve and master the tools you use.
Side note: isort. Tool that automatically sorts your imports: https://pypi.python.org/pypi/isort
Some optional flake8 plugins: flake8-copyright, flake8-exact-pin, flake8-naming, flake8-docstrings, flake8-diff, flake8-tuples.
How about manual Quality Assurance? Checking a responsive design is probably something you need to do by hand, or spend time learning some new tools.
Single, one-time cleanups? Fine. But then also add tools to check that the package stays cleaned up, otherwise you keep spending your time on this issue.
Should a pep8 error end with a CI failure? If you have decided that you want to adhere to pep8: yes!
Before QA, you should do some pre-QA. Use bobtemplates.plone to create new Plone packages.
What kind of things can you check? Tests, coverage, plone.recipe.codeanalysis, state of translations, documentation syntax on PyPI, working documentation references and typos, a proper MANIFEST.in to avoid 'brown bag' releases (missing files), undeclared dependencies, accessibility, check javascript on major browsers and screen sizes, commit messages, no XXX or TODO markers in the code, have a change log entry for each fix or feature.
When do you run them? You can do it manually. Have a pre-commit hook. Before a release, but don't throw this job at the one doing the release, but do it yourself as initial committer. A per distribution CI job.
Some ideas. Keep the noise level low and targeted: if the whole team needs to look at a ten megabyte CI report, that is not nice. When you work on pull request branches, you can use git commit --amend and git rebase to restrict the change to a few or even just one single commit. Integrate checks in your editor. Maybe goal for Plone 5.1 to have zero code analysis failures, see http://jenkins.plone.org/job/code-analysis. Check the dependencies of packages. Zuul as project gating system: http://docs.openstack.org/infra/zuul/. Moving to Python3, will that ever be possible without best practices? Can a newcomer release Plone 6, or does it take one person who knows all the small details?
Final thoughts:
- No bike shedding, just use linters.
- No manual steps, let CI warn.
- Always room for improvement, fill your toolbox.
- Always have an eye for quality on every action.
Question: is setting up Jenkins difficult? You can look at https://github.com/plone/jenkins.plone.org which is a buildout, with nothing really specific for core Plone, you can use it for yourself. When I started using it at Der Freitag, it was basically half a day of work to setup.
How to enforce the style guide for every single package? Setup a Jenkins job or step to run this check. See the Github repo above.
Eric Steele: The State of Plone
Keynote talk by Eric Steele at the Plone Conference 2015 in Bucharest.
Plone had its 14th birthday last week. Lovingly cared for, with the weight of experience behind it.
First of all, thank you. A lot of work and love has been put into Plone 5. We have built something truly special.
Plone 5 has an all new design. A toolbar on the left for editors, with basically everything you used to be able to do from the edit bar. It lives outside of the theme, so no more problems with a theme that would mess up your edit experience.
Folder contents, with for example bulk adding of tags or changing the state. Upload multiple files.
New TinyMCE, much better than what we had previously. Works responsively, so also on other devices.
We redesigned the control panel. Add-ons page has a complete revamp, much nicer and easier to understand. Social media settings.
Theming has been changed. A Diazo theme as base. You can copy the base theme in the UI and edit it right there. A really powerful tool, it makes the initial theming experience much nicer.
Our default content story is Dexterity now. You can still use Archetypes if you want, for now.
We have finally got recurring events rights, like adding an event that recurs each Sunday until the end of the year.
Plone 5 is accessible, meeting lots of accessibility requirements. You will hear more about that in the keynote tomorrow. New multilingual framework, in the core, working for both Dexterity and Archetypes. So: all the content, all the time, for all the people.
There is lots of room for extending Plone and integrating it into all kinds of other systems.
We have sturdy security, especially with automatic CSRF protection.
Evolution of hackability. In earlier Plones you could work through the web, but this was not considered best practice, because you could not really manage your code nicely. Eric Brehault said: we should build an unhackable core, but add something hackable on top of it. Empower people in all the right places. Customization is a feature. We can use through the web prototyping to rapidly come up with a solution that you can then export.
Annette Lewis is a designer of website on Penn State University. New Plone user. She runs about 40 sites. Base theme and policy products, and the rest is done entirely through the web, with dexterity, diazo, theme editor. "It feels like you can do a lot from the front without touching the web." She now has a two week release cycle for sites.
Mosaic is the successor to the oft-repeated Deco project. Not in core. It allows you to stick content on a page, move it around however you want. Look for plone.app.mosaic. Asko Soukka has been working on this two years straight. In beta. We are close to having it for real. Dexterity plus Diazo plus Mosaic is really powerful.
Thoughts for the future
Expand the api: plone.api and a restful api. plone.api is in core. We have started on the restful api.
Javascript front end. Some people are working with AngularJS, others with RequireJS. We have a history of trying things out, and then coming up with better ways, like Diazo beating Deliverance and XDV. We can do that with Javascript frameworks.
Backend consolidation. Plone solved a csrf vulnerability in Zope last week. So I have tried to stay diplomatic about it, but my current feelings are: get your act together or get out of the way.
When you show people Plone, they go from "wow, are you still around?" to "wow, this rocks!"
Ned Jackson Lovely, the PyCon chair, said: "I consider Pyramid and Plone to be significant enough for inclusion in PyCon."
Of course we are!
zest.releaser 4.0: major update
zest.releaser has a fresh 4.0 release, with some nice improvements. So, what's new?
zest.releaser 4.0 has various improvements. We have better errors and warnings, did some cleanup, added little tricks, and have a recommended list of extra packages to use in combination with zest.releaser: check-manifest, pyroma, wheel, and twine.
Main improvements
Errors and warnings
We are better at showing errors. This is especially true for errors when uploading to PyPI. They were unintentionally swallowed before, so you did not notice when an upload failed. Oops, sorry about that.
Errors and warnings are more noticeable, because we colorize them. Errors are red, warnings are magenta. We use the colorama package for this.
We not only do this for lines that are printed explicitly by zest.releaser itself, but try to do this also for output from programs that we call, like check-manifest and python setup.py. This is a bit tricky though. Program should print standard messages to the standard output file and errors and warnings to the standard error file. Not all do that. So we might be missing some colors.
We allow retrying some commands when there is an error. Currently this is only done for commands that talk to PyPI or another package index. We ask the user if she wants to retry: Yes, no, quit. If for example PyPI upload fails because you have the wrong credentials in your ~/.pypirc, you can edit this file and tell zest.releaser to retry the command.
Cleanup
Python 2.6 is not officially supported anymore. It will probably still work, but we are no longer testing against it, so issues may start creeping in. This says nothing about the packages that are created: zest.releaser on Python 2.7 is still perfectly capable of creating distributions for older Python versions. Or for Python 3.
Sorry, zest.releaser itself does not run on Python 3. At least I have not tried it, and certainly the tests will be a pain to fix for both 2 and 3.
We have removed code for support of collective.sdist. That package was a backport from distutils for Python 2.5 and earlier, which we do not support.
Little tricks
We do not accept y or n as answer for a new version. I saw that with a few packages, which seems an obvious mistake.
When doing a postrelease, we add a always edit the changelog file to get a new version section with the text '- Nothing changed yet'. Now, when you do a prerelease and no real changelog entry has been added, so the text is still there, we warn and ask if you really want a new release. Probably you want to stop, check your version control history, and add one or more proper changelog entries.
zest.releaser makes commits in the prerelease and postrelease phase. Something like Preparing release 1.0 and Back to development: 1.1. You can add extra text to these messages by configuration in your setup.cfg or global ~/.pypirc.
One nice use case for this is telling Travis or Jenkins to skip Continuous Integration builds, like this:
[zest.releaser] extra-message = [ci skip]
This depends on how your testing server is setup. It might not have this enabled, or it might be looking for a different message.
Of course, you can also add your favorite geeky quotes there:
[zest.releaser]
extra-message =
No one expects the Spanish inquisition!
So long and thanks for all the fish.
recommended extra
We have a list of recommended extra packages that we think are helpful for most users of zest.releaser. This is in the zest.releaser[recommended] extra. So if you use pip, you can install those like this:
pip install zest.releaser[recommended]
With buildout, you would use a section like this:
[script] recipe = zc.recipe.egg dependent-scripts = true eggs = zest.releaser[recommended]
What is currently in this list? We have check-manifest, pyroma, wheel, and twine.
check-manifest
This checks if the distribution that zest.releaser will create of your package contains all the files that you expect to be there. In some cases your setup.py will contain enough information for Python to include the correct files. Or you rely on a helper like setuptools-git, but then you miss files when someone else who does not have this helper package makes the next release. So in a lot of cases you will want to use a MANIFEST.in file. As an example, zest.releaser itself currently has this:
recursive-include zest * recursive-include doc * include * exclude .installed.cfg global-exclude *.pyc
You can run check-manifest as a standalone command. When used in combination with zest.releaser, the prerelease part calls the check-manifest command for you. That may look like this:
$ fullrelease INFO: Starting prerelease. Do you want to run check-manifest? (Y/n)? listing source files under version control: 6 files and directories building an sdist: mauritstestpackage-0.2.dev0.tar.gz: 6 files and directories copying source files to a temporary directory building a clean sdist: mauritstestpackage-0.2.dev0.tar.gz: 5 files and directories files in version control do not match the sdist! missing from sdist: CHANGES.rst suggested MANIFEST.in rules: include *.rst MANIFEST.in is not in order. Do you want to continue despite that? (y/N)?
check-manifest may report some files as missing from the source distribution where you know this is fine. You can tell check-manifest to ignore those by adding a setup.cfg file next to your setup.py. ``zest.releaser`` itself currently has this:
[check-manifest]
ignore =
doc/build
doc/build/*
We could add some more there, like bootstrap.py, buildout.cfg, and .travis.yml.
For more info on check-manifest, see its PyPI page: https://pypi.python.org/pypi/check-manifest
pyroma
This does various checks on your package. Most of them are about your setup.py. This results in a rating between zero (bad) and ten (good). The rating is also given with the name of a cheese. If your package 'smells' good, you get a better cheese. Where does the name of the package come from? It checks the aroma of your package: Python + aroma = pyroma.
You can run pyroma as a standalone command. When used in combination with zest.releaser, the prerelease part calls the pyroma command for you. That may look like this:
$ fullrelease
INFO: Starting prerelease.
Do you want to run check-manifest? (Y/n)? n
Run pyroma on the package before tagging? (Y/n)?
INFO: ------------------------------
INFO: Checking /Users/mauritsvanrees/own/mauritstestpackage
INFO: Found mauritstestpackage
INFO: ------------------------------
INFO: Your package does not have classifiers data.
INFO: You should specify what Python versions you support.
INFO: Your package does not have keywords data.
INFO: Your package does not have author_email data.
INFO: Setuptools and Distribute support running tests.
By specifying a test suite, it's easy to find and
run tests both for automated tools and humans.
INFO: ------------------------------
INFO: Final rating: 6/10
INFO: Compté
INFO: ------------------------------
Continue? (Y/n)? n
For more info on pyroma, see its PyPI page: https://pypi.python.org/pypi/pyroma
wheel
There are various package formats for Python. Until now, zest.releaser was only doing source distributions, by calling python setup.py sdist. We still do this. Note that we have never released binary eggs (bdist_egg).
wheels are the shiny new Python package distribution format. zest.releaser 4.0 supports creating them and pushing them to PyPI.
Should you want this? Maybe. See http://pythonwheels.com for deciding whether this is a good idea for your package. Briefly, if it is a pure Python 2 or pure Python 3 package: just do it.
Also, if you are using buildout: sorry, buildout currently (2.3.1) does not support wheels. It is fine to create them, but you should still create a source distribution as well. zest.releaser does that, like it always has.
Say you want zest.releaser to create a wheel. How do you do this? You add a setup.cfg file in the top level directory of your package, so next to setup.py:
[zest.releaser] create-wheel = yes
Or if you know you want this for all your packages, you can also do this globally by adding the same text in your ~/.pypirc.
zest.releaser then takes care of the rest: when releasing, it creates a plain old source distribution and a shiny new wheel and uploads them to the package index.
For more info on wheel, see its PyPI page: https://pypi.python.org/pypi/wheel
twine
Since version 4.0, we prefer twine for uploading to the Python Package Index, because it is safer: it uses https for uploading. If the twine command is available, it is used for uploading to PyPI.
Note that we call the twine command directly. If the twine command is not available, you may need to change your system PATH. In the case of buildout, you may need to say dependent-scripts = true in the section where you add the zest.releaser[recommended] egg.
How does it look? When I used zest.releaser to release itself, it looked like this:
INFO: This package is registered on PyPI. Register and upload to pypi (Y/n)? INFO: Running: twine upload dist/* -r pypi Uploading distributions to https://pypi.python.org/pypi Uploading zest.releaser-4.0-py2-none-any.whl Uploading zest.releaser-4.0.tar.gz
When your package is not registered yet on PyPI, twine will currently fail. So you have to register the package manually. Use your browser to login at PyPI and then use the package registration form.
We could consider letting zest.releaser call the old-style python setup.py register instead, or ask the user. If you have thoughts about this, you can use our issue tracker to start a discussion.
As an aside, did you know that there is a test website for the Python Package Index? You can use this for making test releases of packages. If you want to take zest.releaser for a test run and do not want to publish your package on the real PyPI yet, you can release to https://testpypi.python.org. Edit your ~/.pypirc file to something like this:
[distutils]
index-servers =
pypi
testpypi
[pypi]
# default repository: https://pypi.python.org/pypi
username:maurits
password:secret
[testpypi]
repository:https://testpypi.python.org/pypi
username:maurits
password:secret
Then when running our fullrelease or release command, answer 'no' when zest.releaser asks if you want to upload to pypi and answer 'yes' when asked to upload to testpypi.
For more info on twine, see its PyPI page: https://pypi.python.org/pypi/twine
That's it. I hope you enjoyed reading about the improvements. Now go use it! Get it with pip install zest.releaser or pip install zest.releaser[recommended] at https://pypi.python.org/pypi/ or read more at http://zestreleaser.readthedocs.org
Pieter Hintjens: ZeroMQ
Pieter Hintjens gives the keynote talk about ZeroMQ, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
For ZeroMQ, see http://zeromq.org
Distributed systems are about making software more like the real world. Rule one of building software: bring the community in it.
Underlying trend: the cost of scale. How do you upgrade a billion computers? You can't. The amount of computers in your pocket keeps doubling every few years.
Building culture. Building knowledge.
ZeroMQ is one of the first successful communities that is built around distributed systems. Hundreds of language bindings.
Conway's law: an organisation builds software that looks like the organisation. To build distributed systems, you need to be a distributed organisation. Meetings are barriers for organisation. Including daily stand-ups. You block everyone's time until you have reached a decision. Meeting people, exchanging ideas, is fantastic.
ZeroMQ 2, 3, and 4, from a long time ago, were incompatible. When you break contracts, you are an asshole.
I don't know where my community comes from. I don't have to trust them, don't have to pay them.
You can do anarchism. It does not work. One devious character and the system falls apart. And those people are there. You need to have contracts, enforcement. Ability to participate and to reject participants.
Why has no one in this room created a pull request for ZeroMQ?
What you want, is participation. What kills a system, is that no one cares.
Open source is science. Closed source is magic.
Pizza places have gone bankrupt because they put ansjovis on the wrong pizza. They broke the pizza contract. But I don't care how clean their kitchen is, as long as we don't get sick. The pizza should be a certain size, temperature, price, and then it is fine.
When someone submits a bad pull request, I merge it. I don't care. What happens if someone keeps submitting bad pull request? If it is easy to make a mistake it is also easy to revert it or fix it. Don't release from master, because it may be broken. But push to it all the time, otherwise you break the flow. Community is more important than the code. You always merge someone else's pull request. My daughter is maintainer on some of my projects, see merges my pull requests and I go like 'yeee!' That is important.
You can make money from good open source. Leverage, re-use all the knowledge from ZeroMQ. People are making money from this every single day. Making money is still difficult. Open source opens doors, also for single people with large organisations as clients.
Once you start thinking about money, your brain gets wired for cheating, being dishonest.
Anyone can solve problems, but they are often the wrong problems. Closed source is like that. Paying people for writing software is like that.
Most open source still fails, it is not a magical formula. But the best will emerge.
Oscar Vilaplana: Orchestrating Python projects using CoreOS
Oscar Vilaplana talks about Orchestrating Python projects using CoreOS, at PyGrunn.
See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.
This is about orchestrating stuff. Not specifically Python, but the examples will be with Python.
Reliability. Your app does not care where it is running. Runs locally, runs on the cloud, same thing. Portability, repeatable. Loose coupling, compose micro services, makes it easier to scale them, mix and extend.
Cluster-first mentality. Even development machines can run in clusters. Lot of different containers, servers, ports, all connected, how do you manage this. Do you need to be developer and operations officers in one? Let the system figure it out. Let other smart people figure out and determine where the containers are.
One deployment tool for any service, makes them better and better, new service is then not much extra work.
Demo: deploy db, flask app, scale them. RethinkDB, 3 replicas. Give a name to the service, connect to it from the app by name. With kubectl start the service with the wanted amount of replicas.
CoreOS: Kernel + Docker + etcd. Read-only root, no package manager, self auto-updating, systemd for setting limits to a service and making sure it starts and restarts when it is stopped. Socket activation, starting a service only when someone starts using it.
etcd is a distributed configuration store. Atomic compare and swap, change this setting from 1 to 2, otherwise fail. HTTP API. Configurable at runtime. etcd set /pygrunn/current_year 2015
fleet is a distributed systemd. Start a service somewhere in a cluster. Coordination across the cluster. Rules like do not start A and B on same server because they are cpu hungry.
Service discovery: ambassador pattern, talk to someone who knows where the services are.
flannel does something like that. Per-cluster and per-machine subnet. Every container has an ip on a subnet.
Kubernetes: process manager. A pod is a unit of scheduling, name, container image, resources. Labels for finding things.
Replication controllers: create a pod from a template. Ensure that the exact number of pods are running. Upgrades.
Service discovery for pods. An ip per service.
Demo.
It took me a while to wrap my head around it. Look at the slides in the calm of your home.
Me on Twitter: https://twitter.com/grimborg
And see http://oscarvilaplana.cat