First I noted this blog entry by Tom Lazar. Two simple lines in a .pdbrc file in your home directory are enough to give you tab completion in your pdb session:

    import rlcompleter
    pdb.Pdb.complete = rlcompleter.Completer(locals()).complete

Be sure to read the comments at Toms site. Also see the original recipe at ASPN.

I tried this and it worked perfectly. That should give me a productivity boost. Since I work with Zope I started a bin/zopectl debug session and tried tab completion there, but that didn't work. That's logical as the zope debug is not the same as a pdb session. Daniel Nouri came with the obvious answer: just do import pdb; pdb.set_trace() in your debug session! Works fine then.

But then I thought: if this is possible in the pdb, then it should surely be possible in a normal python session. So I searched and found an article by Jacob on ParTecs Weblog. This shows that you can add a .pythonrc.py file in your home directory which contains:

    import rlcompleter, readline
    readline.parse_and_bind('tab: complete')

Set an environment variable:

    export PYTHONSTARTUP=~/.pythonrc.py

Now startup python and you have tab completion. Grand! Jacob's article shows another handy function that you can put in that file, so be sure to read it. While you are at it, check out some more pdbrc ideas. I added some lines there as well.

This still doesn't give you tab completion in a zopectl session though, unless you do the pdb trick. But I prefer not to type/paste that everytime I do some zopectl debugging. I guess zopectl doesn't even try loading that .pythonrc.py file, probably for safety reasons, so noone can sneak in some code. Let's see if we can find a workaround.

Hm, the $PYTHONSTARTUP environment variable is only read for interactive python shells. Apparently a debug session is not considered interactive.

Okay, for starters I can shave off a few key strokes. Run bin/zopectl debug and then:

    import user

This will run your ~/.pythonrc.py file. Two notes:

• This does not use the $PYTHONSTART environment variable. It is hardcoded to load only the .pythonrc.py file in your home directory. So if you can't or don't want to set that environment variable, you can still just do an import user in an interactive python session and your .pythonrc.py file will get loaded.
• If you have defined a function in your file, it is not available for use directly like in a really interactive python session, but only with the module name attached; so with the mentioned example file, you can need to type user.source(user), which shows the source of the user module.

Now my reasoning goes like this. The .pythonrc.py file is meant for interactive python sessions. A zopectl debug session sure feels like an interactive python session. In fact, it is an interactive python session, as zopectl debug uses the -i option to let python execute some statements and then drop into in interactive mode; but that is a special case, so your user file does not get loaded. But the general idea remains: it would be handy to load those user defaults. Of course that would only be handy in debug mode, not in other zopectl modes.

How do we do this? We change the $SOFTWARE_HOME/Zope2/Startup/zopectl.py file:

    def do_debug(self, arg):
        cmdline = self.get_startup_cmd(self.options.python + ' -i',
                                       'import user;' +
                                       'import Zope2; app=Zope2.app()')
        print ('Starting debugger (the name "app" is bound to the top-level '
               'Zope object)')
        os.system(cmdline)

The addition of import user is the only change there. For reference, the command line then becomes (split into a few lines here):

    /usr/bin/python2.3 -i -c "from Zope2 import configure;\
    configure(\'/home/maurits/instances/eXtremeManagement/etc/zope.conf\');\
    import user;import Zope2; app=Zope2.app()"

Since we do this in the do_debug function, this only gets used when you do a zopectl debug, not in a zopectl start or fg. So it should be safe, as far as I can tell.

Actually, after rereading I see that Jacob put something very similar into his above linked article.

So how about it? Can this go into Zope?

I can imagine that there are not too many people who know both conferences. At least I wore the T-shirt from EuroPython at New Wine and that only got blank looks. A short introduction seems in order then. Python is a programming language; I use it daily for my job. EuroPython is a yearly conference in Europe for python users and developers, sometimes affectionately called pythonistas. New Wine is a Christian movement that originated in England and has made the leap to the Netherlands and other countries. Their goal is: equipping churches to see Jesus' kingdom grow. Last week I visited the Dutch Summer Conference for the third year in a row.

As an aside: I haven't quite finished adding to this website summaries of all the talks at EuroPython that I went to. I plan to do that eventually, but first I want to migrate my home grown Zope 2 weblog to Plone plus Quills. If you don't know what I am talking about, then you are probably more in the New Wine camp than in the EuroPython camp. ;-) Anyway, you can expect more EuroPython stuff to appear here and also more New Wine summaries, although I will probably do those in Dutch only. I'll try to make it easier to view just the English weblog entries for those of you who are just as good in Dutch as the English speakers at the New Wine conference. :) Right, on to the comparison.

There are of course lots of similarities between the two conferences. You learn a lot, you meet new people, you get to know friends better, there is a good atmosphere and you have a lot of fun.

There are also marked differences:

• At EuroPython you get to know python better; at New Wine you get to know God better.
• At EuroPython no people got healed as far as I know; at New Wine several people got healed. I know at least of three people who got their foot healed; one person's back has improved considerably; one little girl who got bullied at school the whole year was now filled with joy.
• At EuroPython you learn to program the way Guido indented it (note for New Winers: that's not a spelling error); at New Wine you learn to live the way God intended it (note for pythonistas: that's also not a spelling error). I know I made some happy progress here, thanks to a good talk with Michelle and a seminar by Paulien Zeeman. All glory to God though.

By the way, Guido and God are much alike: they are both Benevolent Dictators For Life. For God life is a bit longer though. Oh, and they are both Dutch. ;)

If forced to choose, I would pick the New Wine Summer Conference any day. Except in winter, as that wouldn't make much sense. ;) Preferably though I would again visit them both next year.

Premature optimization is the root of all evil - C.A.R. Hoare

• Slow startup times are okay if you hardly ever start the program.
• Speed may not be important for a nightly cronjob.
• Actual user experience: does the program feel slow?

Plan:

1. Get the architecture right
2. if bugs: fix them
3. while code is too slow:
   • find worst bottleneck by profiling
   • try to optimize, running unit tests
   • if not faster: undo last code changes

Bottlenecks:

• maybe just your processor, memory, network, I/O, database problem
• Use OS tools, e.g. time, top, dstat, gkrellm, xosview to see in which program the problem is.
• python tools: profile (cprofile in python 2.5), hotshot, print statements

Big O notation

• differentiates between slower and faster algorithms, when the dataset imcreases.

Try to avoid O(n^2) and slower algorithms for large sizes of n.

Performance may be less important than code readability and maintainability.

Strategies:

• change algorithms
• change the architecture
• avoid nested loops
• move loop-invariant code out of the loop
• update only changed data
• cache instead of recompute or reload (but this can be error-prone)
• conversely, keep things out of the cache, as this might exhaust memory and start eating into disk space.
• Use multithreading for I/O bound code

File operations optimization:

• read file completely and then process if it is small
• read and process line by line if it is large
• use database instead of flat files

Python specific:

• Use python -O
• avoid exec and eval
• avoid from module import *
• shortcut namespace searches (e.g. opj = os.path.join)
• change code to use C-coded objects: lists, tuples, dictionaries, sets.

Don't waste developer time on unnecessary optimizations.

Independent Zope and Python consultant. Using Zope and Python since 1999.

Python 2.4 gives string templating, which is better than e.g. print statements.

Quixote - PTL:

import quixote quixote.enable_ptl()

Positives:

• allows template re-use

Negatives:

• broken html possible
• what would an html-only person think?

Zope - DTML:

• It is a generic scripting language
• Highly tied to acquisition
• Tell the html writers to leave the dtml variable alone and they can do the rest.

Positives:

• simple templates
• not just xml/html

Negatives:

• acquisition
• one big namespace
• not good with dreamweaver and friends

Zope - ZPT

Well known in the Zope community.

Positives: - Clean namespaces

• limits you from introducing business logic in templates
• source is valid xml/html

Negatives:

• 2 or 3 new languages
• macros are confusing, especially nesting them
• very tied to Zope
• tied to generating xml/html

Cheetah

• not limited to xml
• not tied to one framework

Positives:

• familiar, python-like
• compact
• caching

Negatives:

• not enough python-like, so yet another language
• we're not helping the web monkeys

Nevow: really aimed at twisted.nevow

Positives:

• Clean separation of data, logic and presentation
• does help the web monkeys

Negatives:

• very verbose
• meant for Twisted and yes it is twisted!

Others: - Myghty

• Preppy
• XSLT
• PyMeld, meld2, meld3
• Kid attribute language like ZPT

Types: preprocessor, class based, attribute languages, DOM based

We want to:

• Replace attributes, values and tags.
• Repeat tags
• Remove attributes, values, tags and node
• html quoting

What if we did that?

• simple
• no new languages
• work with raw html

Twiddler

• n = t.getById(Something)
• n.replace(value, tag, **attributes)
• n.repeat(value, tag, **attributes)
• n.remove()
• Put code in html comments, mostly at the top.
• t.clone() for reusing common material
• Use name attributes:

Positives:

• works with real html
• no new language
• simple as possible

Negatives:

• verbose
• not battle-proven
• maybe slow

I hadn't seen meld3 when i started developing Twiddler, but they are scarily similar. It uses meld:id attribute.

But it's not just about html. Think about sending an email. Or a css file. Or things that are just not xml.

What about i18n, html quoting, stx, rst? We can do that.

See the presentation.

Lambda is not going away, so don't worry about that. (Applause)

Python 3000 philosophy

• Don't design a new language. No pet peaves like adding a switch statement, redoing the if statement or whatever.
• Fix early design bugs.
• Allow incompatible changes (within reason). We're not maximizing breakage of course.
• Get rid of (morally) deprecated features. Old style classes are no longer the default.

Python 3000 process:

• We have to make selections. Too many proposals compete for time.
• Maintaining python 2 is a big issue. Back porting may be partly possible.

Release Schedule

• First alpha: not before next year.
• Final release: probably a year after that
• 3.1 and 3.2 may follow soon after for fairly small but logical changes.
• 2.6 will come, 2.7 likely, 2.9 is as far as we'll go.

How incompatible can it be?

• new keywords allowed
• dict.keys(), range(), zip() won't return lists. This kills dict.iterkeys(), xrange(), itertools.izip()
• All strings will be unicode. Mutable bytes datatype, represented as a character array underneath. For strings always use encoding.
• Binary file I/O redesign. Use bytes objects when reading from or writing to files.
• Drop '
• as alias for !='.
• But not:
  • dict.keys instead of dict.keys()
  • change meaning of else-clause of for/while
  • change operator priorities

How to migrate code?

• Perfect mechanical translation will not be possible. Many changes are semantic rather than syntactic. So a live programmer will need to look at it. Tools like pychecker can help though.
• Most likely approach:
  • Use pychecker-like tools to do an 80 percent job.
  • Create python 2 version that warns about dead ends.

Python 3000 will not:

• have macros, programmable syntax, etc
• add syntax for parallelization (use zip())

Python 3000 features: read PEP 3100 for a large list. See the peps

Basic cleanup:

• kill classic classes
• all exceptions must derive from BaseException
• int/int will return a float
• remove last differences between int and long
• kill:
  • sys.exc_type()
  • dict.has_key() (use in)
  • file.xreadlines() and actually file and readlines()
  • apply(), input(), buffer()

Minor syntactics:

• exec becomes a function again
• kill `x` in favor of repr(x)
• range() will behave like xrange()
• xrange() dies
• zip() becomes izip(0
• lambda lives! The lambda lovers tried for a year to come up with a better version, but failed. But it has its uses, so it can stay.

String types reform:

• bytes and str instead of str and unicode
• maybe bitwise operations on bytes, but that might add too much to to API.
• All data is either binary or text.

New standard I/O stack

• C stdio has too many problems
• bytes/str gives an opportunity to fix all this.

Print becomes a function!

• print x, y, z becomes print(x, y, z)
• print >> f, x, y, z becomes print(x, y, z, file=f)
• This makes it much easier to switch use logging.
• Skip the space or newline? Use printf() or printraw() or whatever we come up with.

Drop default inequalities: <= and friends are not useful between unrelated objects of different types. '== and !=' should always be possible.

Thanks to the Google team for T-shirts and sponsoring!

If it is possible to make a 2.x version that is somehow compatible with 3.0 then that would be nice.