Weblog
Elizabeth Leddy: Unloading Plone: Approaching Scalability in Integrated Plone Systems
By Elizabeth Leddy, United States
While there is an abundance of documentation on ways to achieve better performance with Plone, there is nothing quite like actually doing it, especially with other system components getting in the way. This case based look at the performance and scaling of Plone as part of an integrated system will cover perceived front-end latency, system stability as related to Plone responsiveness, and how to set up a hardware forward architecture. This talk is meant for designers and integrators of large Plone installations.
Plone is slow and that is fine, we just have to deal with it. We are not Drupal. Scalability is talent plus hardware. You are going to require a lot of hardware.
Measure. If you don't know what is going on, you don't know how to act. Use tools like Munin. It records all activities on your system over time. Monit restarts your stuff when it goes down, or sends emails. Ganglia is an alternative to munin. Zope Health Watcher, has munin plugin.
Add whale watchers. Are triggered by system anomalies and collect info about the system at that time.
Shard you zeo: split it into several little zodbs with mount points; at least to this for the portal_catalog. You don't want to move a 300GB Data.fs. Tune your zodb caches. Watch out: do not copy stuff from one shard to another; you will lose stuff.
You want to use 50 percent of your system on average. That way you can add another zope instance quickly when you get a traffic hit.
2 zopes per cpu. The system is better at context switching than zope, so you probably want just one or maybe two threads per zeo client.
See if you can do some processing at another time, so not when saving an object.
Serve static content from a static web server. Apache is better at server all those icons than Zope.
Use css sprites.
Use HAProxy, it is really, really awesome! It has graceful reloads, it knows if a zeo client is ready with a previous request, you can configure a warmup time (give zope the time to startup without hammering it with requests).
When you display data from a different server (like SQL) add tal:on-error to display something else when it goes wrong.
For third party services, do socket.settimeout(2) (if using python lower than 2.6).
Restarting Zope is not a sin. Use monit to restart zope when it takes up too much memory.
A packed zodb is a happy zodb. Do it for your system, it will love you.
On backups:
- use repozo for backing up
- get some sweet disks to make the zodb faster
- think about when you are going to pack your zodb, don't do it when everyone is busy using your server
Suggestions from the room: zeoraid, crickets, zenoss, munin.zope.
Wichert Akkerman: Euphorie: combining grok, dexterity sql content in a single application
By Wichert Akkerman, Simplon, The Netherlands
A case study describing the Euphorie application. Euphorie combines many different types of technology in a single Plone site: dexterity content types, SQL stored content, custom traversal, sphinx documentation and even a grok application.
Euphorie: operational risk assessment tool. Identify risk in your organization and how to take them away. Does this at a European level. Developed by Syslab, TNO, Simplon (myself). Ambitious projects. 27 languages, 20 countries.
We wanted something that is easy to learn, use and extend, maintainable, scalable. Clean, documented, tested code.
It was not a very Plonish problem. So we unploned plone: no redirect tracking, no content rules, no sharing tab. Fairly simple to do.
Content types: archetypes has several problems, one of which is that it is slow. So we used dexterity, which has fewer problems.
In this project we do some very interesting things with Plone. One is dexterity. Another is the input we receive. A lot comes from Word documents, or the user added styles or linebreaks himself. So we had to do some cleanup work there.
HiFi (high fidelity) prototyping for the design in static pages. It at least allows you to share the css and javascript between the prototype and the actual implementation.
Grok-in-Plone. We basically built a grok site inside Plone.
We had to render our content in three different phaes. For the same risks we needed different forms. First: do we run this risk, yes or no; then: how likely is this; then: what are you going to do about it. So we did some url tricks by inserting 'PathGhosts' in the acquisition chain that added a string to the url although the actual object behind the url was still the same. So that got us saner urls.
Session content. The zodb is not fast at writing lots of content. But that was needed for storing the answers the user gave so far in the survey. So we used SQL here, using SQLAlchemy. And we put the user objects in SQL as well.
For documentation we added sphinx to all the packages.
Plone can be fun if you step out of the box.
Wicher Akkerman: Lessons from other frameworks
By Wichert Akkerman, Simplon, The Netherlands
New python frameworks such as pylons, bfg, grok, django take some different approaches than the Zope2 model Plone is based on. This talk will give a brief overview of other frameworks and the lessons that can be learned from them.
Templates
With Zope and Plone you use Zope Page Templates (ZPT). The other frameworks use something similar. ZPT is smarter than you, which is not always handy. Chameleon and Genshi are alternatives used by the others. You can use Chameleon is Plone too now. Deliverance is interesting in overcoming some of the ZPT limits, but it has its limits as well.
Authentication and authorization
Zope/Plone have PAS, the Pluggable Authentication Service. bfg and grok just have something simple here, which is also simple to understand and is usually enough. You could use repoze.who and repoze.what, but that again has too much policy that you then have to override again.
Forms
In Plone you have about five ways to do forms. That is just too much. Several aspects: schema definition, validation, marshalling, generation. Just do one aspect in one form package. Pylons uses formencode, which does no html generation and does not create an object in your database. Other options: formish, repoze.formapi.
Data
Plone uses content types to store in the zodb; CMF content types, archetypes, dexterity, perhaps devilstick. Other frameworks: either zodb objects or sql. bfg can do both. Pylons defaults to SQL, but can do other storage as well, like CouchDB. Storing something into the ZODB has to fit your data model, then it is fine.
Configuration
Plone: partly in zcml, partly in zope.conf, partly in the zodb, for example using xml (GenericSetup). Such a split does not help. The others don't have that. bfg only uses zcml to wire things together, not for configuration; it uses an ini file.
URLs
Zope: everything is based on traversal. The publisher goes through your zodb to look for objects matching the url. Works very well. But then you get other notations, like /folder/@@view, /++resource++package/public.css. There is a lot of complexity there. Even building the urls can be tricky. Other frameworks use routes. You define a set of rules that map urls to code. If you want /register in English and /registreren in Dutch you can do that with routes. It doesn't work that well when you have unstructured data like in Plone.
Documentation
Goal for me: I don't want to learn a magic trick, I want to understand it, understand why this trick is the right way to do it.
- ZTK: lots of documentation about how to develop on top of the ZTK, but not about the ZTK itself.
- Grok: has tutorials and a partial reference; for complex stuff you need to learn more as it is based on the ZTK. Not versioned.
- bfg is good here. It has tutorials, reference, fully versioned documentation.
- Pylons has tutorials, a good reference, but it builds on other packages which do not have good documentation. Pylons people are giving documentation back to those projects now. No versioned documentation.
- Django has everything you want here.
- Plone: lots of tutorials, reference docs need work, a lot of packages on pypi just have one line of documentation, although the documentation of zc.buildout on the http://plone.org site is actually better than the official zc.buildout documentation; it makes you understand what happens. Documentation is not versioned, or at least it is not clearly visible.
You want to be in control of your code, be able to understand it. At least you have the source code with all these frameworks.
Lessons
- Modularity helps, but can also confuse. Django is not modular enough, but everything is consistent. With the ZTK you need to learn several ways of doing things.
- Being explicit can be better than a framework. Who completely understands Plone portlets, Plone navigation.
- Less features, not more.
- Sugarcoating is not a solution. At some point the nastyness underneath still surfaces.
Jordan Baker: Plone Testing Tools and Techniques
Tutorial by Jordan Baker, Scryent, Canada
Geared towards developers this presentation walks you through developing a simple Plone 3 product using a test-driven or test-first approach. Techniques covered include unit testing, PloneTestCase, test layers, mocking and more. This is an updated and expanded version of the talk "Hone Your TestFu" given at the Plone Symposium East 2009.
I am using Plone since version 1.0. Senior consultant and founder of Scryent, in Toronto, Canada.
Slides are at http://tinyurl.com/pttt-slides
You don't always have to test. Sometimes you are just doing a prototype. But in my opinion: if you write production code, you should write tests. It improves code quality; reduces defects; makes your code more maintainable; it nudges you into more modular, loosely coupled code as that is easier to test; it keeps you focused: if the tests pass then you are done; when you have good test coverage, you can refactor with confidence.
Test first: write the test first, then start coding. It forces you to think about requirements up front. And manual testing is slow and error-prone.
Test levels: functional/system (black box), integration (your code in combination with Plone), unit (just your code).
More tests is not always better. Those tests need to be maintained too. With too many tests, it may start taking too long to run all the tests. Focus on lots of lower level tests as they are fast to run.
Unit tests: written from the developer's perspective. Test your components in isolation; use mocking techniques when you need to test integration with other components.
When working on a particular problem, try running only the immediately relevant tests. Try bin/instance test --help for some options you can pass. When those tests pass, run all tests again to check for unexpected failures.
Integration tests: test the integration of your code with Plone. Can you create the content types you made in the actual Plone Site? Do the correct values end up in the portal_catalog? Are the extra roles you defined actually added when you install your product?
Recently I started using collective.testcaselayer to simplify test setup. It simplifies repetitive Plone testing boiler plate.
Add it in your setup.py in extras_require and tests_require. Then in the eggs of your buildout add your.package[tests].
Use roadrunner to speed up testing: it sets up the layers (the Plone Site) and keeps that available so the setup is much much faster the next time you run those tests. There are some problems with for example z3c.autoinclude. I want to sprint on roadrunner this weekend.
Doctests look like interactive python sessions with documentation/explanation added.
Functional and system tests: end-to-end tests, black box tests, often zope.testbrowser tests, or with Twill or Selenium. Perhaps smoke tests: add specific tests for the most important paths that are used in your Plone Site. Twill is a domain specific language for performing web tests. It also provides a python API.
Use mocking to e.g. patch the time.time method, or urllib, or the MailHost. Martin Aspeli created plone.mocktestcase.
Coverage: answers the question: are all code paths executed in the tests. Does not guarantee that there are no bugs in the code. Use z3c.coverage to get html reports of the coverage.
Write tests at the lowest lelel possible. Take your time, write out what you are trying to do in English first. Is it too hard to test, then simplify your code, split it out.
I want to have an open space on testing.
Rok Garbas: Complex Forms with z3c.form
By Rok Garbas, Slovenia
Rok Garbas talks about the z3c.form form framework. Everybody is talking about is, using it, so it is time you learn how to (ab)use it and step into the Plone future. Tricks and things you might want to hear again. I won't speak about the internal structure, or why you would want to use it, or compare it with other frameworks.
For code examples, see his talk once it is uploaded to the Plone Conference 2009 website.
On the form you can set ignoreContext and ignoreRequest to control whether the form displays any values and where it gets it from.
Don't forget to package="plone.app.z3c.form" /> in your zcml when you use z3c.form in Plone.
You can register zope vocabularies and use that in your forms.
You can change the form layout. For that you create a layout factory, inheriting from plone.z3cform.templates.ZopeTwoFormTemplateFactory.
You can define your own widgets, for example to combine three inputs into one field.
You can create a custom data converter. It converts from the field to the widget (e.g. split fullname field into first name and last name inputs in the widget) or the other way around.
Within the form normally the default widgets are used. If you want to use your own widgets, you need to override the widgetFactory for that field in the form definition.
You can create group forms, which can be compared to Archetypes forms with tabs. Inherit from z3c.form.group.GroupForm.
In form wizards you can combine steps, so you can link different forms. Use collective.z3cform.wizard.
For more information, see http://docs.zope.org/z3c.form and http://pypi.python.org/pypi/plone.app.z3cform
Read Rok Garbas blog.