First I noted this blog entry by Tom Lazar. Two simple lines in a .pdbrc file in your home directory are enough to give you tab completion in your pdb session:

    import rlcompleter
    pdb.Pdb.complete = rlcompleter.Completer(locals()).complete

Be sure to read the comments at Toms site. Also see the original recipe at ASPN.

I tried this and it worked perfectly. That should give me a productivity boost. Since I work with Zope I started a bin/zopectl debug session and tried tab completion there, but that didn't work. That's logical as the zope debug is not the same as a pdb session. Daniel Nouri came with the obvious answer: just do import pdb; pdb.set_trace() in your debug session! Works fine then.

But then I thought: if this is possible in the pdb, then it should surely be possible in a normal python session. So I searched and found an article by Jacob on ParTecs Weblog. This shows that you can add a .pythonrc.py file in your home directory which contains:

    import rlcompleter, readline
    readline.parse_and_bind('tab: complete')

Set an environment variable:

    export PYTHONSTARTUP=~/.pythonrc.py

Now startup python and you have tab completion. Grand! Jacob's article shows another handy function that you can put in that file, so be sure to read it. While you are at it, check out some more pdbrc ideas. I added some lines there as well.

This still doesn't give you tab completion in a zopectl session though, unless you do the pdb trick. But I prefer not to type/paste that everytime I do some zopectl debugging. I guess zopectl doesn't even try loading that .pythonrc.py file, probably for safety reasons, so noone can sneak in some code. Let's see if we can find a workaround.

Hm, the $PYTHONSTARTUP environment variable is only read for interactive python shells. Apparently a debug session is not considered interactive.

Okay, for starters I can shave off a few key strokes. Run bin/zopectl debug and then:

    import user

This will run your ~/.pythonrc.py file. Two notes:

• This does not use the $PYTHONSTART environment variable. It is hardcoded to load only the .pythonrc.py file in your home directory. So if you can't or don't want to set that environment variable, you can still just do an import user in an interactive python session and your .pythonrc.py file will get loaded.
• If you have defined a function in your file, it is not available for use directly like in a really interactive python session, but only with the module name attached; so with the mentioned example file, you can need to type user.source(user), which shows the source of the user module.

Now my reasoning goes like this. The .pythonrc.py file is meant for interactive python sessions. A zopectl debug session sure feels like an interactive python session. In fact, it is an interactive python session, as zopectl debug uses the -i option to let python execute some statements and then drop into in interactive mode; but that is a special case, so your user file does not get loaded. But the general idea remains: it would be handy to load those user defaults. Of course that would only be handy in debug mode, not in other zopectl modes.

How do we do this? We change the $SOFTWARE_HOME/Zope2/Startup/zopectl.py file:

    def do_debug(self, arg):
        cmdline = self.get_startup_cmd(self.options.python + ' -i',
                                       'import user;' +
                                       'import Zope2; app=Zope2.app()')
        print ('Starting debugger (the name "app" is bound to the top-level '
               'Zope object)')
        os.system(cmdline)

The addition of import user is the only change there. For reference, the command line then becomes (split into a few lines here):

    /usr/bin/python2.3 -i -c "from Zope2 import configure;\
    configure(\'/home/maurits/instances/eXtremeManagement/etc/zope.conf\');\
    import user;import Zope2; app=Zope2.app()"

Since we do this in the do_debug function, this only gets used when you do a zopectl debug, not in a zopectl start or fg. So it should be safe, as far as I can tell.

Actually, after rereading I see that Jacob put something very similar into his above linked article.

So how about it? Can this go into Zope?

Premature optimization is the root of all evil - C.A.R. Hoare

• Slow startup times are okay if you hardly ever start the program.
• Speed may not be important for a nightly cronjob.
• Actual user experience: does the program feel slow?

Plan:

1. Get the architecture right
2. if bugs: fix them
3. while code is too slow:
   • find worst bottleneck by profiling
   • try to optimize, running unit tests
   • if not faster: undo last code changes

Bottlenecks:

• maybe just your processor, memory, network, I/O, database problem
• Use OS tools, e.g. time, top, dstat, gkrellm, xosview to see in which program the problem is.
• python tools: profile (cprofile in python 2.5), hotshot, print statements

Big O notation

• differentiates between slower and faster algorithms, when the dataset imcreases.

Try to avoid O(n^2) and slower algorithms for large sizes of n.

Performance may be less important than code readability and maintainability.

Strategies:

• change algorithms
• change the architecture
• avoid nested loops
• move loop-invariant code out of the loop
• update only changed data
• cache instead of recompute or reload (but this can be error-prone)
• conversely, keep things out of the cache, as this might exhaust memory and start eating into disk space.
• Use multithreading for I/O bound code

File operations optimization:

• read file completely and then process if it is small
• read and process line by line if it is large
• use database instead of flat files

Python specific:

• Use python -O
• avoid exec and eval
• avoid from module import *
• shortcut namespace searches (e.g. opj = os.path.join)
• change code to use C-coded objects: lists, tuples, dictionaries, sets.

Don't waste developer time on unnecessary optimizations.

Independent Zope and Python consultant. Using Zope and Python since 1999.

Python 2.4 gives string templating, which is better than e.g. print statements.

Quixote - PTL:

import quixote quixote.enable_ptl()

Positives:

• allows template re-use

Negatives:

• broken html possible
• what would an html-only person think?

Zope - DTML:

• It is a generic scripting language
• Highly tied to acquisition
• Tell the html writers to leave the dtml variable alone and they can do the rest.

Positives:

• simple templates
• not just xml/html

Negatives:

• acquisition
• one big namespace
• not good with dreamweaver and friends

Zope - ZPT

Well known in the Zope community.

Positives: - Clean namespaces

• limits you from introducing business logic in templates
• source is valid xml/html

Negatives:

• 2 or 3 new languages
• macros are confusing, especially nesting them
• very tied to Zope
• tied to generating xml/html

Cheetah

• not limited to xml
• not tied to one framework

Positives:

• familiar, python-like
• compact
• caching

Negatives:

• not enough python-like, so yet another language
• we're not helping the web monkeys

Nevow: really aimed at twisted.nevow

Positives:

• Clean separation of data, logic and presentation
• does help the web monkeys

Negatives:

• very verbose
• meant for Twisted and yes it is twisted!

Others: - Myghty

• Preppy
• XSLT
• PyMeld, meld2, meld3
• Kid attribute language like ZPT

Types: preprocessor, class based, attribute languages, DOM based

We want to:

• Replace attributes, values and tags.
• Repeat tags
• Remove attributes, values, tags and node
• html quoting

What if we did that?

• simple
• no new languages
• work with raw html

Twiddler

• n = t.getById(Something)
• n.replace(value, tag, **attributes)
• n.repeat(value, tag, **attributes)
• n.remove()
• Put code in html comments, mostly at the top.
• t.clone() for reusing common material
• Use name attributes:

Positives:

• works with real html
• no new language
• simple as possible

Negatives:

• verbose
• not battle-proven
• maybe slow

I hadn't seen meld3 when i started developing Twiddler, but they are scarily similar. It uses meld:id attribute.

But it's not just about html. Think about sending an email. Or a css file. Or things that are just not xml.

What about i18n, html quoting, stx, rst? We can do that.

See the presentation.

Lambda is not going away, so don't worry about that. (Applause)

Python 3000 philosophy

• Don't design a new language. No pet peaves like adding a switch statement, redoing the if statement or whatever.
• Fix early design bugs.
• Allow incompatible changes (within reason). We're not maximizing breakage of course.
• Get rid of (morally) deprecated features. Old style classes are no longer the default.

Python 3000 process:

• We have to make selections. Too many proposals compete for time.
• Maintaining python 2 is a big issue. Back porting may be partly possible.

Release Schedule

• First alpha: not before next year.
• Final release: probably a year after that
• 3.1 and 3.2 may follow soon after for fairly small but logical changes.
• 2.6 will come, 2.7 likely, 2.9 is as far as we'll go.

How incompatible can it be?

• new keywords allowed
• dict.keys(), range(), zip() won't return lists. This kills dict.iterkeys(), xrange(), itertools.izip()
• All strings will be unicode. Mutable bytes datatype, represented as a character array underneath. For strings always use encoding.
• Binary file I/O redesign. Use bytes objects when reading from or writing to files.
• Drop '
• as alias for !='.
• But not:
  • dict.keys instead of dict.keys()
  • change meaning of else-clause of for/while
  • change operator priorities

How to migrate code?

• Perfect mechanical translation will not be possible. Many changes are semantic rather than syntactic. So a live programmer will need to look at it. Tools like pychecker can help though.
• Most likely approach:
  • Use pychecker-like tools to do an 80 percent job.
  • Create python 2 version that warns about dead ends.

Python 3000 will not:

• have macros, programmable syntax, etc
• add syntax for parallelization (use zip())

Python 3000 features: read PEP 3100 for a large list. See the peps

Basic cleanup:

• kill classic classes
• all exceptions must derive from BaseException
• int/int will return a float
• remove last differences between int and long
• kill:
  • sys.exc_type()
  • dict.has_key() (use in)
  • file.xreadlines() and actually file and readlines()
  • apply(), input(), buffer()

Minor syntactics:

• exec becomes a function again
• kill `x` in favor of repr(x)
• range() will behave like xrange()
• xrange() dies
• zip() becomes izip(0
• lambda lives! The lambda lovers tried for a year to come up with a better version, but failed. But it has its uses, so it can stay.

String types reform:

• bytes and str instead of str and unicode
• maybe bitwise operations on bytes, but that might add too much to to API.
• All data is either binary or text.

New standard I/O stack

• C stdio has too many problems
• bytes/str gives an opportunity to fix all this.

Print becomes a function!

• print x, y, z becomes print(x, y, z)
• print >> f, x, y, z becomes print(x, y, z, file=f)
• This makes it much easier to switch use logging.
• Skip the space or newline? Use printf() or printraw() or whatever we come up with.

Drop default inequalities: <= and friends are not useful between unrelated objects of different types. '== and !=' should always be possible.

Thanks to the Google team for T-shirts and sponsoring!

If it is possible to make a 2.x version that is somehow compatible with 3.0 then that would be nice.

The Zope Foundation was legally formed in March 2006. Initial board since June: Aroldo Souza-Leite, Martijn Faassen, Rob Page and Jim Fulton.

We have strategic developers, consumers, solution providers, associate members and board members. It is a non-profit organisation.

Purpose: advance the community.

Practical: hold the copyright, trademark, manage zope.org.

The foundation is you! The community is doing the work and should keep the responsibility. We'll try to help and not get in your way. The foundation helps make things easier for others.

Initial committees: membership, finance, Zope management organisation and the www.zope.org web site. We need more volunteers on all committees.

Q: What services should the Zope Foundation offer to developers of the Zope platform?

Martijn: We have subversion, mailing lists, wikis, issue tracker. Not all our services should necessarily be based on Zope, e.g. an issue tracker need not have been made in Zope.

Aroldo: it is a frame. We seek commitment between parties. More meta technical and legal. It is a movement.

Jim: I don't want to be a Zope Pope anymore and I am not. I tried to add a page for a talk i held earlier and I had to manually add a Folder as Manager on the zope site, which feels a bit dirty. So we need something better there.

Christian Theune: +1 on better bug trackers.

Martijn: Help people to not build Zope specific software if it can instead be used outside of Zope. Right now many people don't even try it.

Q: What should the Zope Foundation do for people with an interest in Zope?

Martijn: Give people clearly the impression that Zope and its projects are there to stay, that they are stable. Give them peace of mind when using it.

Aroldo: The Zope Foundation could be a frame for evangelising. A good place to collaborate with other institutions, for example the python.org organisation. We should take care of documentation.

Jim: +1 on all that.

Chris: How are new members invited?

Jim: Dieter has been invited.

Martijn: We had a board decision for that.

Jim: We'll add some info on the site.

Christian: There should be info on where and when conferences are, which companies in the neighbourhood can help customers.

Mark Hamilton: Make the role of Zope clear. People may now step in at either the python or the plone level without really realizing what the Zope level is about.

Several in the room: A sort of Zope newsletter would be good, maybe some development news? There is Planet Zope with an accompanying blog and a planetzope del.icio.us tag. Done by Philipp von Weitershausen?

Jim + Martijn: There are several Zope related projects: zope2, zope3, Five (well, part of zope2), zodb. Maybe we need to split those.

Aroldo: the Zope Foundation can help you decide if you want to join the main Zope effort or start a separate project.

Room: more centralised support for small projects. More info about those projects on www.zope.org, instead of just a subversion repository.

Jim: I agree.

Room: I like the idea of Popes for several projects actually.

Kit Blake: When I talk to a customer, it helps to have a Zope Foundation, because I can point them to a web site. The web site may be ugly, as long as it gives good info, a good image. A Zope Foundation with a good image is good and then I don't mind a messy www.zope.org.

Geir: A neat www.zope.org is good, but not necessarily a task for the Zope Foundation.

Martijn: Whose job would it then be?

Geir: It could be done without the Zope Foundation. The community can do that.

Jim: The Zope foundation owns the domain name, but we don't have the man power to handle the web site.

Tarek Ziade: How about more marketing people?

Martijn: We have to use the right marketing to the right market. So more marketing people are helpful.

Jim: When more than one person steps forward, we'll worry about how to coordinate marketing efforts.

Jim: Do we want to split Zope into more projects? So give some people far more authority and responsibility for e.g. zodb or zope3? Andreas doesn't have the authority to decide where Zope2 is going to be heading.

Russ: You are looking to spread out the load. There is no money available. Only recognition now, so you need to do that part well.

Martijn: Giving credit to people is important.

Christian: Right now, if somebody wants to modify something, maybe large changes that I am unhappy about, then I have no authority to do anything about it. So splitting could help there.

Steve: There are a few roles for a project: technical officer, security officer, some more. These may be the same person. Knowing who has these roles up front would help. Make it clear. Also: define compatibility. What versions is a software program compatible with.

Martijn: Maybe you can help?

Steve: I'm busy applying to become a member. :)

Jim: We are in the bootstrapping phase. I'm not sure of the requirements for new members, but we don't want to turn anyone away.

Q: What shall we do with www.zope.org?

Chris Withers: rm -rf /

Martijn: We have enough ideas, we just need some time. Identity pages for projects are the most important. We should look at TurboGears and Django, because they are doing this well. We should go low tech. Just simple pages. Steve brought up LaunchPad, so we may want to use that. Maybe just use Apache. Anything that works!

Room: Applause!

Martijn: It is a very important part of our effort.

Chris: How do I get in?

Jim: Ask Martijn.

Chris: Okay, I want in.

Christian: Okay, you are in. :)

Martijn: There are ideas, but there are some practical problems, like access to the machines that host zope.org.

Jim: Zope Corp is still hosting that. Okay, we have a few volunteers. Maybe everyone who is still left in the room here? :) We have them on film. :) We have the authority, we just need people to get involved.