Steve McMahon - Plone Production Deployment: Secrets & Tricks

Steve McMahon talks about Plone Production Deployment: Secrets & Tricks, at the Plone conference 2012.

We are going to have a special version of this talk: no secrets, to tricks!

If you are hear, or reading this blog item, you should be a (Unixy) SysAdmin or Friend of a SysAdmin. This is not going to be an advanced talk. If you want something more advanced, find Martin over in the Disco room.

A PHP programmer may run away screaming when you mention reverse proxying and caching, but if they manage a big site, they will use those technologies too, and will not still be on a five dollar/euro per month server.

A stand-alone is great for development, but you should not use it on production. It lacks versatility, SSL support, and more.

So we add a web server, like Apache or nginX. Modern web servers can be used for reverse proxying. It can handle SSL, efficiently queue requests, handle URL rewrites. They generally have battle-hardened logging and request sanitizing. Lots of people have tried to break it. It probably rejects requests that are obviously meant as an attack. It will play well with other server parts.

You may add other web applications. Plone is your CMS, but you may let another application handle one special part of the site. Single Sign On can be done between the two applications.

Donald Knuth famously said: "Premature optimization is the root of all evil." It was expanded by Elizabeth Leddy: "Measure, measure, measure. Then act." Do not make your stack more advanced than it needs to be. Measure, experiment, measure. If the measurement does not show an improvement, reverse the changes. You may not need load balancing, whatever that person on irc tells you.

Leonardo da Vinci said: "Load balancing means always having a canon ready to fire." We replace canons by zeo clients that can handle a request. Such a ZEO setup can scale to multiple machines. It is a good option, even without load balancing: you can use a second zeo client to handle debugging (hope you do not need it) and to run scripts that take long. The zeo server has different requirements for memory, disk space, CPU, so you can optimize. And of course you regularly do a bin/zeopack on the command line to pack your database.

Good load balancers distribute the load very well over the zeo clients that are live. This helps to avoid service interruptions. The balancing scheme can matter. Default is round robin. If sticky sessions are into effect, you usually end up at the same zeo client all the time, which helps for the caches that this zeo client has. But you can change your balancing scheme and measure if that improves responsiveness and stability.

Someone around 1566 said: "Good caching makes light work." You can add a server cache, a reverse proxy cache. Typically this weill be between the web server and the load balancer. It can be on a separate machine. You cache in memory or on disk. A good cache is usually varnish or (poor you) squid. David Glick, during the Cioppino 2012 sprint, said: "Caching is hard." If you expect a lot of traffic for a day, then you may need more aggressive caching."

plone.app.caching is how to do caching in Plone 4. There is cache invalidation, but that is hard. A page may show a portlet that shows a collection, which may need to be invalidated at random times, not having to do anything really with that page, which may not have changed this year. My general strategy is to start conservative with caching and then slowly turn it up. You handle anonymous and authenticated requests differently. Dylan Jay said during his lightning talk yesterday that he simply cached one minute. You do not need an invalidation scheme then.

Choosing some tools. Take care of your tools and they will take care of you. You can do the best of breed approach:

• web server: probably Nginx, maybe Apache
• proxy cache: our community chooses varnish. It makes incredibly good use of memory. It will make really good decisions about memory and swapping.
• load balancer: I love HAProxy. It allows you to use several different schemes. Sticky sessions are a bit tricky to setup, but it can be done. SSL can be a problem, but latest version has improvements.

A perfectly reasonable alternative in some circumstances, is to let the web server do it all. William of Occam, translated to modern speak, said: "Keep it simple, stupid." Having everything in one web server may make that web server too complex though.

Read about hosting here: http://collective-docs.readthedocs.org/en/latest/hosting/ You can edit that document, or propose changes.

Elizabeth Leddy: "Backup is serious shit." Restoring is serious too. You need to test a restore scheme, otherwise you do not actually have a backup. There are horror stories about that. So what you do is: backup, rsync to another server, restore. Use collective.recipe.backup.

Eric Steele: "Rotate your logs, or your server will do." Florian: "At some point, move the old logs as well, as they still keep up space." sudo apt-get install logrotate. Rotate your Z2.log, event.log (instance/zeoclient.log), do it weekly, postrotate do a kill -USR2 on the zope process so the logs are reopened. On the topic of log files: read your event logs. They have important information. Some tracebacks you may know about, and they drown out the nasty stuff. Read them regularly. Try to get the noise in the logs down. Look at mailinglogger for this, which is already in zope, or Sentry. Maybe set the log level to warning instead of info. That helps a bit for performance too.

ZODB packing: use the bin/zeopack script, installed by plone.recipe.zeoserver. Set it up in a cronjob.

Elizabeth Leddy, Cioppino 2012: "These days, I use system packages as much as possible. You should, too." I want my routine security updates of the operating system to apply without me needing to do much about it. The least used feature of the UnifiedInstaller: --with-python=/usr/bin/python2.7. If it meets the requirements, it will use it.

Last advice: never run buildout as root. Always create a separate user and run buildout and run zope as that user. If hostile code ended up in a setup.py of a package and you run buildout as root, then your server is owned by others.

Read about deployment here: http://collective-docs.readthedocs.org/en/latest/deployment/ You can edit that document, or propose changes.

If you edit this, put the complex information in the hosting docs and the beginner documentation in the deployment docs. And own that document as a community.