The past, present, and future of how we test Plone

"Untested code is broken code" according to Martin Aspen and Philipp von Weitershausen. But tested code is broken code too: even if all tests pass, it is not a guarantee that everything works, even with 100 percent test coverage.

So why do we write tests?

• Avoid regressions
• Improve your confidence when you change the system, for example from Python 2 to Python 3.
• Really: to minimise the chance of being ashamed.

History

When did the first tests get into Plone? I got to October 7 2002. A kind of continuous integration was done first in 2004. Golden age of Plone testing was between say 2009 and 2013, when PloneTestCase was there, plone.app.testing was created as new version, there was a Testing Team, Jenkins was added in 2010. Travis CI was used since 2012, a lot of add-ons started using that. We got robotframework around 2012, for end user testing in the browser, using Selenium. In 2013 mr.roboto was born so we could make sure that when one package changed, the full test suite of all Plone was run.

In 2018 GitHub Actions started. In 2019 Cypress could be used as alternative for robotframework, Volto is using this. Around now, some tools around code analysis have gotten updated.

Our current process

• You make a pull request.
• GitHub Actions checks are run, mostly code analysis, in some cases building docs.
• With a comment on GitHub you can start Jenkins jobs.
• When everything is green, the PR can be merged

Future

These are some ideas, I will be asking questions.

• Are we using the right tools? Most of the Python community uses pytest, we use zope.testrunner. You can actually use gocept.pytestlayer now so your Plone testing layers are treated as pytest layers. You need to move the Python code into src. And don't mix doctests and unit tests in the same module.
• Browser testing: Selenium? Cypress? Playwright?
• Move from Jenkins to GitHub Actions? Thank you Gil Forcada for keeping Jenkins up and running so far!
• Can the tests go faster? Any easy wins? With parallelisation on my M1 I ran the tests in four minutes.
• Go to a monorepo? For things that are used outside of Plone: no. But it may make a lot of things easier. Recently I changed the minimum password length and needed changes in eight packages or so; with a mono repo this would have been much easier.
• Do Classic UI and Volto test the same features? Could the plone.restapi tests run on Guillotina or Nick?
• Which version are we testing? We are not testing the backend of Plone against Volto until it is too late.
• Do we have the right tests? 350 acceptance tests (robot plus cypress), 800 integration tests (Python), 8000 unit tests (Python plus Jest)
• What else could we be testing? Accessibility testing. Visual regression testing. Performance tests. Type checking. In VS Code: go to Python, Analysis, Type Checking Mode, and set it to basic.
• Error tracking
• Distributed tracing for seeing how long things take to execute.

Audience:

• The Classic UI tests could be split from the REST api tests.
• For pytest, you could move the tests out of the package code to the root.
• Could we make pytest the default for add-ons?
• Mono repo does not solve everything. We need to think about architecture. But fewer repos is good.