Fast tests

Slow tests suck. They are annoying and slow down you and your team. Hours and hours of engineer time is lost waiting for the CI to finish. I'll go through a number of approaches and principles to help you write fast tests.

I started plone.api way back, now in core. I have organised about twenty sprints. Next up is the Nix(OS) sprint 21 to 25 November in Lanzarote.

I developed an app for security on Mac, see https://paretosecurity.com/. We had tests for this, they were fast, but after a while I checked again and it took 3 minutes for only 300 unit tests. What? We need to bring this down. Long tests can cost money, and at least they cost developer time. If the CI pipeline takes twenty minutes, then you cannot wait for it, and you cannot start anything new in that time. There are a few things you can try.

Try one approach, measure it locally, measure it on CI, merge when it is an improvement, wait a few days before trying the next thing.

Use hyperfine to compare results of commands.

Running tests

Throw money at the problem: make sure your team has fast laptops and that you fast have CI runners. Buy a Mac mini for less than 1000 euros and use it as a fast CI runner in GitHub or GitLab.

Try pytest --collect-only. Ballpark: should take 1 second for 1000 tests. If it takes longer, there is something wrong. You are probably telling it to look in too many directories.

Try pytest --collect-only --noconftest. Does this differ much with the previous result? 

export PYTHONDONTWRITEBYTECODE=1

Usually we know a few tests that are slow. You can mark them as slow, and do not run them by default, only if you run them explicitly. Then do run them on CI.

Try pytest-incremental or pytest-testmon.

Writing tests

Make sure tests are not doing any network I/O, like to GitHub or gravatar. Use pytest-socket to get them to fail, and then fix them by mocking the network access.

Use pyfakefs to use a fake filesystem in memory.

Do all tests need a database? If tests only do simple things, like doing some calculations, do not setup a database. Or maybe not all tables are needed for this test.

Can I create the database only once? At the end of a test, you can truncate tables, so you don't need to recreate them.

Can I populate the database only once?

Parallellisation

The big wins are here, but it is more involved, so start with the others first.

pytest-xdist helps here. Make sure your parallel runs don't influence each other, like writing to the database when another process truncates it.

pytest-split is good for the CI, not for local runs.

Now that your tests are fast, how do you keep them fast? We made BlueRacer.io. Free for per personal use and open source software.

Extra tip: run pytest --lastfailed to only run the tests that failed in the last run.

See https://github.com/zupo/awesome-pytest-speedup for the full list with more information.

Audience: gocept.pytestlayer helps for running Plone tests in pytest.