Alexander Loechel: Modern Python Testing

I have worked in the German air force. They have rules on how to write software. For example, you start with user requirements. In open source, who is writing down user requirements? The requirements are there, but they are often implicit. Write tests that test the requirements.

'Testing leads to failure. Failure leads to understanding.'

A test is a specific set of assertions.

You can test requirements, design, interfaces, code/implementation, documentation (embedded code), conventions.

The basis for tests in Python is the unittest module. It has lots of specific assertions, like assertEqual, assertIsNotNone, etc. But why can't we simply use assert? That is what the pytest module does instead. It makes writing the test simpler. It is by now the de facto standard in the Python world. It has a pluggable add-on system.

There is unittest2, nose, nose2, but today they are mostly outdated. Use pytest.

Robot framework is used by Plone to do web testing with a real browser.

Richard Feynman: 'The first principle is that you must not fool yourself, and you are the easiest person to fool.'

For test runners we have the unittest testrunner, zope.testrunner, pytest-testrunner, gocept.pytestlayer. Such a runner collects tests for execution and shows the outcome to the user. If you use unittest, you can use these. A test runner can interact with other tools, like coverage.

On a command line you can usually run python setup.py test to run tests. Or you have scripts to run it, like buildout can install with a recipe.

You want to run your tests automatically on a test server:

• Travis CI has Linux and MacOSX machines, and are perfect for pure Python tests
• Circle CI has Docker containers for Linux and MacOSX
• Appveyor tests on Windows.

On Travis you can use travis_retry in your .travis.yml file to retry a command three times. This can help when there is a temporary network problem.

tox is a test invocation tool. You use it to run your tests on multiple environments, for example Python 2 and 3, or with an extra package installed. You can use additional helpers like pyenv for virtual environments. One of the environments can apply isort, or run zest.releaser, even if these environments are not run by default. Or you can run a linter and have it report on your code quality.

My wishes for better practices in Plone:

• Adapt tox on all packages.
• Switch to different package structure and enforce that, like bobtemplates.plone, with docs, src, Tests.

Maybe do not ship with the tests in the PyPI releases: this code will not be run in production. Tests belong in the source distribution, but not in binary packages like wheels or eggs. The unit tests should be tested within the actual package. Integration tests could live somewhere else.

See http://plone-best-practices-discussion.readthedocs.io/

See the slides.