Gil Forcada Codinachs: Beyond QA

Look at these Python packages: pep8, pyflakes, flake8, bobtemplates.plone, plone.recipe.codeanalysis. Sysadmins like to be lazy, or should be lazy: he does not want to get a call at midnight to get back to work putting out a fire, because everything is running smoothly, and anyone can do it because everything is setup correctly.

From one colleague I used to need to check all his work, even after explaining, explaining, explaining. There might be tests, maybe not, maybe failing, non standard code, etcetera.

Instead of spending time on real reviewing, it turned into a fight about how to do things, which style to use. So I stopped complaining. I did it differently: let the machines work for you. Let continuous integration tell you all the things.

Are tests passing? Pull requests on Plone are passed through our Jenkins test server.

Does code adhere to a style guide? Is it automatically checked, also by Jenkins or Travis?

QA is about asking why? Why did you make the changes this way? Why did you create this new buildout part? Step back from the line-to-line review and get the full overview. Maybe the change should be in a new package, or the change should not be done at all.

Side note: NASA's take on software development. See http://www.fastcompany.com/28121/they-write-right-stuff

End user or developer documentation. Maybe you create a new option for a buildout recipe, but nobody knows about it unless you document it. Maybe you write everything in English and not everyone can read it. Does it run in all browsers, did you test that? Maybe there is a broken layout, because you forgot to add an upgrade step. Is a dependency missing?

Side note: for checking layouts, see huxley, screenster, PhantomCSS.

So how do we fix it?

Have a style guide and conventions, some documentation on how your team wants the code to look like, what best practices you should adhere to. The best style guide is a coded guide, an automated check, especially as a commit hook. Use the plone.recipe.codeanalysis[recommended] recipe. Use flake8, not only in this recipe, but look at it yourself, integrate it in your editor, look at all the plugins that have been created for it. Maybe create one of your own, it is not hard.

Use a CI (continuous integration) system, it saves time. It is a newcomer's best friend, or even the seasoned veteran's best friend.

Fill your toolbox with zest.releaser, z3c.dependencychecker, lots more. Look for tools, share tools and experiences that help you adhere to the style guides and best practices. Improve and master the tools you use.

Side note: isort. Tool that automatically sorts your imports: https://pypi.python.org/pypi/isort

Some optional flake8 plugins: flake8-copyright, flake8-exact-pin, flake8-naming, flake8-docstrings, flake8-diff, flake8-tuples.

How about manual Quality Assurance? Checking a responsive design is probably something you need to do by hand, or spend time learning some new tools.

Single, one-time cleanups? Fine. But then also add tools to check that the package stays cleaned up, otherwise you keep spending your time on this issue.

Should a pep8 error end with a CI failure? If you have decided that you want to adhere to pep8: yes!

Before QA, you should do some pre-QA. Use bobtemplates.plone to create new Plone packages.

What kind of things can you check? Tests, coverage, plone.recipe.codeanalysis, state of translations, documentation syntax on PyPI, working documentation references and typos, a proper MANIFEST.in to avoid 'brown bag' releases (missing files), undeclared dependencies, accessibility, check javascript on major browsers and screen sizes, commit messages, no XXX or TODO markers in the code, have a change log entry for each fix or feature.

When do you run them? You can do it manually. Have a pre-commit hook. Before a release, but don't throw this job at the one doing the release, but do it yourself as initial committer. A per distribution CI job.

Some ideas. Keep the noise level low and targeted: if the whole team needs to look at a ten megabyte CI report, that is not nice. When you work on pull request branches, you can use git commit --amend and git rebase to restrict the change to a few or even just one single commit. Integrate checks in your editor. Maybe goal for Plone 5.1 to have zero code analysis failures, see http://jenkins.plone.org/job/code-analysis. Check the dependencies of packages. Zuul as project gating system: http://docs.openstack.org/infra/zuul/. Moving to Python3, will that ever be possible without best practices? Can a newcomer release Plone 6, or does it take one person who knows all the small details?

Final thoughts:

• No bike shedding, just use linters.
• No manual steps, let CI warn.
• Always room for improvement, fill your toolbox.
• Always have an eye for quality on every action.

Question: is setting up Jenkins difficult? You can look at https://github.com/plone/jenkins.plone.org which is a buildout, with nothing really specific for core Plone, you can use it for yourself. When I started using it at Der Freitag, it was basically half a day of work to setup.

How to enforce the style guide for every single package? Setup a Jenkins job or step to run this check. See the Github repo above.