Plone conference 2007, Naples. Speaker: Christopher Johnson

This talk is about how Plone GetPaid is being developed. We made a name for how we did that: Social Source.

Developers are often not too happy about input from others. "We know quite well what we need to do, thank you."

How you get it done determines what you get done.

We saw that Plone was flexible and useful out of the box. But payment through Plone was not possible, so you had to use other products. We wanted to do something about that.

I am an entrepreneur, not a developer, and I did not know much about e-commerce when I started.

Step 1. So we started looking: what is already out there? Why do we need something else. We wrote about that on the website of GetPaid: http://plonegetpaid.com.

Step 2. Then we made a plan. What do you do and how? Who is going to benefit? And make the site pretty.

Step 3. We recruited leaders and participants.

Step 4. We refined the requirements. We got input from users, developers, UI experts.

Step 5. We asked for money. If you do not ask, you will not get it. How to ask? Connect needs with value: how does it help them? Be transparent about what you do with the money. Be patient and persistent.

Step 6. We celebrate successes. Reward and recognize people. Communicate about recent developments in the project: what has been done?

Step 7. Sustain it: have fun together, motivate people.

We had three sprints so far. The code is at Google Code. We have a blog, mailing list, irc channel #getpaid.

Perhaps Plone could benefit from a process like this? Where is Plone going? What is our vision? Developers and users should answer that question. [Someone from the audience says: Alexander Limi is going to announce something about that this week.]

Plone conference 2007, Naples. Speaker: Gael Le Mignot Company: Pilot Systems

Why this talk? Plone is powerful but slow. Us developers tend to forget about optimization. "Computers are powerful, right?" Yes, but Plone can be slow, so we need to look at that. You can do several things.

• Use fast algorithms, especially for code that is called often. In a test algorithm going from list to set speeds things up by a factor of 2700.
• Trust the base python code, instead of rolling your own functions.
• Compute once, use often. (So use plone.memoize for instance.)

How do you know how much time your code takes?

• Write a decorator printing the time taken by its function execution.
• On Unix systems, use resource.getrusage
  • Should you look at system or user time? user time is the time spent in your python/zope code.
  • Unix does not give you per-thread accounting. So you could use just one thread during testing. But that steers you away from real life.
• Due to caching etcetera a second test will probably give different results.
  • Fresh load strategy: test on a freshly started zope
  • Second test strategy: run the test twice and throw away the first result.
• DeadlockDebugger: meant for debugging deadlocks. But you can also use it for profiling.
• While running the benchmarks, do not surf the web or otherwise use your computer.

Write efficient Zope code:

• Use the catalog, but do not store too much in it, else it gets too big.
• Page templates and python scripts are slow.
• Content types, tools and python code in general are faster.
• Use caching with decorators for slow methods. Do not store zope objects in the cache, but simple python types. Think about what you put in the cache key. Roles? User ids?
• Think about expiry of the cache, otherwise your memory will run full. Options:
  • keep it for a limited time
  • store a maximum number of objects
  • use LRU (or scoring)
  • restart Zope
  • Have sme button you can push to flush the cache.

Pilot Systems is releasing GenericCache, a GPLed pure python cache, so you may want to look at that.

Conflict errors:

• With threads and transactions, nothing is written to the ZODB until the end. So when your transaction takes too long, another transaction may commit earlier, which causes you code to run again, slowing you down even more.
• Try not to change too many objects in the same transaction.
• But look out for resulting consistency errors. Look for a good point to commit the transaction which will not leave your data in an inconsistent state.
• Some conflicts can be resolved. For instance: for the most recent access date you can just pick the most recent. Use the _p_resolveConflict method for those cases. It takes old state, saved state and current state as input. It is up to you to use it.

Memory freeing may kick in too late, wasting memory that you know can be freed. This is made difficult with several layers on top of each other: python garbage collector, C libraries, operating system.

Swapping memory: the size is no real problem. The problem is the frequency with which things are fetched from disk to memory.

• Do I have a memory problem? Use vmstat 2 on the command line to look at your virtual memory state. Interesting columns in its output: si and so on Linux, pi` and po on FreeBSD.
• Use the gc (garbage collection) module of python. Get a list of objects, check for uncollectable cycles, ask for manual collection.
• Monkey patch malloc. Track malloc/free calls with ltrace. In the GNU C library write hooks to malloc.

Optimizing memory:

• Use the catalog, like mentioned already. It is better for your memory than awakening objects.
• Store big files in the filesystem. This will avoid polluting the ZODB cache. Even better: use Apache to serve them directly. Use tramline. The BlobFile directory can be served by Apache.
• You can use del to manually remove large objects before running some slow code.

Massive site deployments:

• Use ZEO. Easy to set up. Makes Zope parallellizable on many CPUs which makes Zope faster.
• But: it will slow ZODB access, especially when doing it over a network. Also, the ZODB is on one server, which can become a bottle neck.
• Use a proxy-cache like squid. This works best for anonymous users. Caching for logged-in users is more difficult.
• Squid varying: cache based on e.g. the language that the browser includes in the request.
• Create a second ZODB on a different server by exporting the site as static html or with a zexp or just copying the ZODB. Syncing back is tricky, but we did it with PloneGossip and SignableEvent.

Conclusion: Plone is not doomed to be slow! But optimization has to be part of all steps, from design to code. Fast CPUs and large memory does not solve this problem.

For the most recent version focusing on Plone 4, see my talk at the Plone Conference 2012.

For a version from 2010 focusing on Plone 3.3 and 4, see a different article on my blog.

How should you handle translations now when dealing with locales, i18n, products, packages and Plone 3.0?

For clarity: I use the following terms here: a product is in the Products dir, a package is in the lib/python dir.

Please correct me if anything I write here is wrong.

i18n/mydomain.pot
i18n/mydomain-nl.po
i18n/mydomain-de.po

locales/mydomain.pot
locales/nl/LC_MESSAGES/mydomain.po
locales/de/LC_MESSAGES/mydomain.po

i18ndude rebuild-pot --pot locales/mydomain.pot --create mydomain .

i18ndude sync --pot locales/mydomain.pot locales/*/LC_MESSAGES/mydomain.po

# Compile po files
for lang in $(find locales -mindepth 1 -maxdepth 1 -type d); do
    if test -d $lang/LC_MESSAGES; then
        msgfmt -o $lang/LC_MESSAGES/${PRODUCTNAME}.mo $lang/LC_MESSAGES/${PRODUCTNAME}.po
    fi
done

Door Duco Dokter van Goldmund, Wildebeast & Wunderliebe op de Nederlandse Plone gebruikersdag, 19 september 2007.