Testing tutorial: http://plone.org/documentation/tutorial/testing

Look at the tests in Plone itself, or at third party products. Find a product that does something similar and look at its tests.

If a product is not tested at all, a good way to start is add testbrowser tests to just create a content type of that product.

Use pdb in doctests: http://pypi.python.org/pypi/doctestpdbhacks

The p4a packages have good doctests.

The danger of doctests is that you get neither good documentation nor good tests.

Patrick Gerken: Grease monkey script for hiding code examples and showing more links on pypi: http://userscripts.org/scripts/show/50614

The paster templates could add a bit more documentation about how to test.

collective.testcaselayer should help avoid some of the boiler plate code needed.

Hanno created plone.recipe.alltests to run all tests of the packages you selected.

z3c.testsetup should do the same.

You can use nose for tests.

In a pdb in your tests you can start a zope server where you can use your browser to click through the site:

>>> from Testing.ZopeTestCase.utils import startZServer
>>> startZServer()
('127.0.0.1', 55143)

When a customer has installed 25 extra products and wants me to migrate his site, I want to be able to give him a good idea of how hard that is going to be, how much trouble, without needing to be convincing. Do the products have tests? Is there good test coverage?

On pypi you can see how many times a package has been downloaded. Gives an indication.

I want at least some automatic information, like number of tests, coverage.

It takes about one or two hours per package to configure a buildbot. You can grab statistics from that. Maybe automatically run pyflakes and pep8 tests.

Each package has an owner.

We can calculate a value for a package, like lines of code times the coverage. Can be touchy.

Perhaps create a directory buildbot in the collective, where people can put buildout config files for their packages.

It is better to have a src directory in a package instead of having the code next to the setup.py file.

Add automatically collected information to PloneSoftwareCenter.

Could be tricky to prevent someone uploading code to the collective that is then executed by the buildbot and for instance starts sending spam.

Show translation coverage.

We should not do rating (yet).

Look at the pylint package.

You can make pyflakes less complaining.

pep8 and pyflakes tests could be optional.

Action points, we can sprint on this:

• get buildbot infrastructure
• add fields to PSC for code coverage, and number of tests; could be annotations
• POST request for writing the info there from the buildbot
• scripts to parse the buildbot output and post the info

Plone 3.3 had about 6 months of prereleases. Very little testing and feedback coming in, very late. Shouting "Please test!" did not help. If companies were using prereleases and it was working fine you only heard it when you explicitly asked it.

Several browser tests are missing, like for collections, which is why they were broken for two releases. Also Selenium tests.

Hanno was looking at all the bugs, reassigning them. Very useful, but too much for one single person.

Another task would be specifically testing just before or after a release.

Having data about how many times the versions.cfg of a release candidate is downloaded would be handy. Getting no negative feedback but also no positive feedback is not good. Are there no problems or is no-one using it.

Having a list of things to do before a release is handy.

Testing a release candidate during the Plone TuneUps is good. Make some noise about that.

InstallFest: get people to try out the installers and be available on irc for support.

http://pypi.python.org/pypi/mr.ripley: replay the requests of a live site on a staging server.

Idea: button "Ready for Plone 4" for third party products on plone.org?

Buildbots for the most popular third party products?

Important things to do:

1. Triaging bugs.
2. Get people to test releases.

Tell Joanna when a new release is coming up so we could use testers in the Plone TuneUp.

Get a list of which things to change when a new release is there, like also the topic of the #plone irc channel.

Test coverage runs for packages are interesting as well.

Each zeo client has its own object cache. If you have an 8 GB Data.fs in the zeo server, 16 GB memory, and 8 zeo clients, then the clients each have only 2 GB memory available. If something is needed that is outside the cache you need to access the disk, which is bad.

The bfg folks made something where you have to explicitly mark an object for writing; so objects that are just for reading can be shared between clients.

You can use the multiprocess package.

You can cache calculated results with memoize. Is then stored in every zeoclient. memoize is ready to switch to memcached so you can share the memoized data between the clients.