Steve McMahon talks about Plone Production Deployment: Secrets & Tricks, at the Plone conference 2012.

We are going to have a special version of this talk: no secrets, to tricks!

If you are hear, or reading this blog item, you should be a (Unixy) SysAdmin or Friend of a SysAdmin. This is not going to be an advanced talk. If you want something more advanced, find Martin over in the Disco room.

A PHP programmer may run away screaming when you mention reverse proxying and caching, but if they manage a big site, they will use those technologies too, and will not still be on a five dollar/euro per month server.

A stand-alone is great for development, but you should not use it on production. It lacks versatility, SSL support, and more.

So we add a web server, like Apache or nginX. Modern web servers can be used for reverse proxying. It can handle SSL, efficiently queue requests, handle URL rewrites. They generally have battle-hardened logging and request sanitizing. Lots of people have tried to break it. It probably rejects requests that are obviously meant as an attack. It will play well with other server parts.

You may add other web applications. Plone is your CMS, but you may let another application handle one special part of the site. Single Sign On can be done between the two applications.

Donald Knuth famously said: "Premature optimization is the root of all evil." It was expanded by Elizabeth Leddy: "Measure, measure, measure. Then act." Do not make your stack more advanced than it needs to be. Measure, experiment, measure. If the measurement does not show an improvement, reverse the changes. You may not need load balancing, whatever that person on irc tells you.

Leonardo da Vinci said: "Load balancing means always having a canon ready to fire." We replace canons by zeo clients that can handle a request. Such a ZEO setup can scale to multiple machines. It is a good option, even without load balancing: you can use a second zeo client to handle debugging (hope you do not need it) and to run scripts that take long. The zeo server has different requirements for memory, disk space, CPU, so you can optimize. And of course you regularly do a bin/zeopack on the command line to pack your database.

Good load balancers distribute the load very well over the zeo clients that are live. This helps to avoid service interruptions. The balancing scheme can matter. Default is round robin. If sticky sessions are into effect, you usually end up at the same zeo client all the time, which helps for the caches that this zeo client has. But you can change your balancing scheme and measure if that improves responsiveness and stability.

Someone around 1566 said: "Good caching makes light work." You can add a server cache, a reverse proxy cache. Typically this weill be between the web server and the load balancer. It can be on a separate machine. You cache in memory or on disk. A good cache is usually varnish or (poor you) squid. David Glick, during the Cioppino 2012 sprint, said: "Caching is hard." If you expect a lot of traffic for a day, then you may need more aggressive caching."

plone.app.caching is how to do caching in Plone 4. There is cache invalidation, but that is hard. A page may show a portlet that shows a collection, which may need to be invalidated at random times, not having to do anything really with that page, which may not have changed this year. My general strategy is to start conservative with caching and then slowly turn it up. You handle anonymous and authenticated requests differently. Dylan Jay said during his lightning talk yesterday that he simply cached one minute. You do not need an invalidation scheme then.

Choosing some tools. Take care of your tools and they will take care of you. You can do the best of breed approach:

• web server: probably Nginx, maybe Apache
• proxy cache: our community chooses varnish. It makes incredibly good use of memory. It will make really good decisions about memory and swapping.
• load balancer: I love HAProxy. It allows you to use several different schemes. Sticky sessions are a bit tricky to setup, but it can be done. SSL can be a problem, but latest version has improvements.

A perfectly reasonable alternative in some circumstances, is to let the web server do it all. William of Occam, translated to modern speak, said: "Keep it simple, stupid." Having everything in one web server may make that web server too complex though.

Read about hosting here: http://collective-docs.readthedocs.org/en/latest/hosting/ You can edit that document, or propose changes.

Elizabeth Leddy: "Backup is serious shit." Restoring is serious too. You need to test a restore scheme, otherwise you do not actually have a backup. There are horror stories about that. So what you do is: backup, rsync to another server, restore. Use collective.recipe.backup.

Eric Steele: "Rotate your logs, or your server will do." Florian: "At some point, move the old logs as well, as they still keep up space." sudo apt-get install logrotate. Rotate your Z2.log, event.log (instance/zeoclient.log), do it weekly, postrotate do a kill -USR2 on the zope process so the logs are reopened. On the topic of log files: read your event logs. They have important information. Some tracebacks you may know about, and they drown out the nasty stuff. Read them regularly. Try to get the noise in the logs down. Look at mailinglogger for this, which is already in zope, or Sentry. Maybe set the log level to warning instead of info. That helps a bit for performance too.

ZODB packing: use the bin/zeopack script, installed by plone.recipe.zeoserver. Set it up in a cronjob.

Elizabeth Leddy, Cioppino 2012: "These days, I use system packages as much as possible. You should, too." I want my routine security updates of the operating system to apply without me needing to do much about it. The least used feature of the UnifiedInstaller: --with-python=/usr/bin/python2.7. If it meets the requirements, it will use it.

Last advice: never run buildout as root. Always create a separate user and run buildout and run zope as that user. If hostile code ended up in a setup.py of a package and you run buildout as root, then your server is owned by others.

Read about deployment here: http://collective-docs.readthedocs.org/en/latest/deployment/ You can edit that document, or propose changes.

If you edit this, put the complex information in the hosting docs and the beginner documentation in the deployment docs. And own that document as a community.

Elizabeth Leddy talks about Plone as a Development Platform during the Plone conference 2012.

This is a highly opinionated talk about the future of Plone from a Framework Team member, developer, project manager, consultant, loud mouth and general advocate of change, also known as Elizabeth Leddy. (Those are her words, but you knew that.)

This is the third time I give this talk this year. Sorry, but each time I do this it gets more positive, which never was the original idea.

In the Plone 4 series there are two trends. We have been and are busy with modernizing the user experience and modernizing the architecture.

A PLIP is a Plone Improvement Proposal. Plone is almost perfect, but people keep wanting to improve it. There is a Roadmap Team, but they do not decide what happens. People still need to give ideas and write code. They do not do that by themselves. So if you have an idea for Plone that you think should be in Plone, go ahead and make a PLIP, also if you are not a core developer.

There are 1132 open tickets on http://dev.plone.org/. We have fixed lots of tickets this year, but this still means 8 tickets per core developer. And we still get 3.4 tickets per day. If you are a core contributor and can fix 4 tickets during the next 12 months, that would be awesome. If you are sprinting this weekend and do not know where to start, grab a ticket! Ask me if you do not know which one to pick.

In January 2012 29 new add-ons were registered on http://plone.org/. The Plone collective on github has 680 repositories, and rising, which is pretty amazing. The number of core packages are actually going down a bit, making things simpler.

What will Plone 5 be? It is whatever the release manager says it is. We will see what gets in it and whether all the shiny new goodness is in it.

There is a bitterness factor in development that keeps Plone 5 too far in the future. Plone 3 introduced a regression. There was less development going on. There was more complexity. Some people were leaving the community. There is a popular stackoverflow question from 2008 that basically asks: "I may want to use Plone, but is the steep learning curve worth it?"

I want to focus more on making the easy things easy. Keep the syntax simple. plone.api is a great start. Reduce the number of files needed to start. Reduce the number of imports needed.

We need feedback and if you are a newbie you are perfect for giving feedback. Documentation is a scapegoat: if we write better APIs, then we need less long documentation. We need developer driven development. Think about that fellow developer that you drank a beer with last night: is he going to be pissed if he reads your code and needs to use it?

Plone 5 has the potential to make happy developers! A thriving community of happy developers makes it easier to hire new people, getting people enthousiastic about working daily with Plone. I usually get some dot NET developers or old Java developers to start working with Plone, as for them Plone is easier. A large community of happy developers also makes it easier to fire someone, or get rid of a client: they can get new developers.

The add-on self certification checklist should get a 'Certified by Spanky' checkbox: can a self-professed average developer like Spanky use this add-on?

Yes, we hear you and we are serious about fixing it. At the Plone Konferenz we started a Plone Pain Top 10. A few of them:

• Getting site root is too hard. Use plone.api.
• Getting a component or view versus getting a tool. Planned in plone.api.
• Cleanup of documentation in core. Help needed.
• Move more code to Python instead of skin templates and scripts.

Plone 5 todo items:

• Make Archetypes optional. Widgets in dexterity should get better. Recreate the default Plone content types in dexterity.
• No more KSS used in core.
• Javascript date formatting.
• plone.api in core

Summary: Plone 4 modernizes the user experience. Plone 5 should modernize the developer experience.

Please decide to commit yourself to Plone the community. Too many people sitting on the fence, waiting for new stuff to land in core, or for the next CMS written in Pyramid, just means we have a busted fence. Nothing happens.

Make it a policy in your company that new employees sign up for Plone on day one. Do the paperwork: get them to sign the Plone contributor agreement, make sure they get a plone.org account. Persist the Plone culture. Pre-install an irc client and configure it to connect to the #plone channel. New developers should not just get to know the code, but also the community. Teach them how and when to file bugs and properly contribute fixes. Do mentorship in your company. You get good developers that way.

Resolve to travel. Hosting a sprint is a great way to contribute back to the community. Ploners love to travel. Send a person to a conference, also non-technical people. You get them to a conference, we get them to stay. Have someone give a talk, for example present a case study. And let them stay to sprint. Form an alliance with other developers or companies. Show what you have done for clients that is not publically available as open source. Maybe you can share.

Resolve to encourage. Test a new version of Plone. Put it on the staging server of a client and test it yourself or even have your client test it. Make a policy to stop forking packages if possible. Have your fixes end up upstream so you do not need your fork. Set aside time for a company sponsored PLIP.

Resolve to sprint this weekend. I will work on two sprints. For beginners: finish the Plone Contributor Agreement process. For advanced developers: goodbye .cpy.

Plone has the potential to make happy developers, but only you have the power to make Plone!

The Plone 5 code name should be: No excuses.

Rok Garbas finally talks about Deco at the Plone conference 2012.

We have been talking about Deco for at least four annual Plone Conferences in a row. I got introduced to it during the Living Statues Sprint here in Arnhem in 2010.

Let's first have a definition for clarity: Deco is a layout editor. Tiles are something different, they can be used outside of Deco.

You need to install the Deco package. You get a toolbar at the top, in an iframe. You add a Deco page and add tiles there, for example just some tiles with rich text, using TinyMCE. You can drag and drop tiles around on the page.

So, the toolbar at the top is an iframe. It should work fine, stretching if needed. With iframe.js this is easy to do, also if you need to put other parts of the DOM in an iframe. We use this to show both the back-end and the front-end on the same page, without needing to worry about how the back-end looks and whether your Diazo theme conflicts with it.

The next thing is tiles. Tiles are mean and lean. If you use Grok instead of zcml, one file is enough. Tiles are a brower view with an extra interface. There are persistent tiles (stored in ZODB, compare portlet settings) and transient tiles.

Then the third part is the real Deco part. The communication between the Deco iframe and the main frame (the front-end page) was hard to get right and was the main reason why Deco took so long to develop.

Current integration in Plone is: Deco Lite. This is a dexterity behavior, so only available for dexterity content types. It works only on the content area. Portlets are still working as before.

If you want to use tiles without Deco, you can install plone.app.tiles.

Deco is really close for production use. A few bugs need to be fixed. I will sprint on it at the end of the conference and hope people will join me.

A few weeks ago I was frustrated with the code, because the Javascript code was untested. Now it is 50 percent tested. If you want to know more about automated testing of Javascript, come to the Sea Sprint talk, later today.

Nejc Zupan talks about plone.api during the Plone conference 2012.

This is a story about how Plone development went to a bit of a dark place and is now coming into the light again.

http://eestec.net is a website for more than 2500 European engineering students. Running in Plone 3. Two of those engineers stayed in Plone. The main reason was that we switched to Plone 3 during development. It got harder to train people. Plone 4 has improvements, but there are still problems.

You always have to remember from where you should import that function or class. Copying or moving objects is hard to remember how to do programmatically. Sample code is harder to read because of this.

So the idea of plone.api was born during the Plone Konferenz in Munich early this year. Work was done during the Belgium Beer Sprint. Eestec students new to Plone were working on rewriting the site in Plone 4. They gave feedback about what should become part of plone.api.

Inspiration came from:

• PEP20 (import this)
• PEP8
• Pareto principle: we document and support the 20 percent of tasks that you use 80 percent of the time.
• Documentation first.

Getting the site root: api.portal.get().

Moving an item: api.content.move(source=..., target=...).

It is documented. We document first. We write narrative documentation. The online documentation is pulled from the code and the comments in the code.

It is tested. We have about 95 percent test coverage. Narrative documentation is included in the tests. We have Continuous Integration set up with Travis CI.

Import and usage style:

from plone import api
portal = api.portal.get()
user = api.user.create(username='bob')

We have api.content, api.user, api.group.

It is used in the wild. There is tutorial.todoapp as a tutorial application you can try out. eestec.portal uses it. We have been using it in production for about a month on a portal with 100k objects and 300 users. No problems, although it is alpha code.

Where should you use it? In integration code? Yes. In add-on code? Likely. In core code? No.

In the future we should get api.role:

with api.role('Manager'):
    # Do something as manager

• api.env.debug_mode(), api.env.plone_version()
• api.system for running upgrades and things like that

We could create plone.jsonapi to package code around json.

We need lots of help.

• We want more translations of the documentation.
• Multilingual Sphinx on readthedocs.org?
• Show us your home-baked maintenance scripts.
• Sprint on this during the conference sprints.

Utopia would be that the Plone learning curve gets lower because of this and we can put up a small cheat sheet for new (or old) developers.

Fight for your rights as a developer!

Download plone.api and give it a spin on your development environment.

Code: https://github.com/plone/plone.api

Documentation: https://readthedocs.org/docs/ploneapi/en/latest/index.html

Steve and Andy: I think we should actually use this in the core code too. At least if a new package is using plone.api and this package gets moved into the core, then we should not require a rewrite without plone.api before it is accepted into the core.

Lightning talks on Wednesday during the Plone conference 2012.