Short summary: it's good book. I learned a thing or two from it. I don't know well it would work as an introductionary text for someone new to unit testing (or Python). Some of the bits seemed overcomplicated and underexplained, parts of the example code/tests seemed to contain design decisions received from mysterious sources.

Incidentally, Packt uses a simple yet effective method for watermarking e-books: my name and street address are displayed in the footer of every page. What's funny is that the two non-ASCII characters in the street name are replaced with question marks. It's not a data entry problem: the website that let me download those books shows my address correctly, so it must be happening somewhere in the PDF production process. I didn't expect this kind of Unicode buggyness from a publisher. Then again there were occasional strange little typographical errors in the text, like not leaving a space in front of an opening parenthesis in an English sentence, or using a never-seen-before +q= operator in Python code. I was also left wondering how the following sentence (page 225) could slip past the editing process:

doctest ignores everything between the Traceback (most recent last call).

Thankfully those small mistakes did not detract from the overall message of the book.

I liked the author's technique of showing subtly incorrect code, letting the reader look at it and miss all the bugs, and then showing how unit or integration tests catch the bugs the reader missed. I'm pretty sure there's at least one remaining bug that the author missed in the example package (storing a schedule doesn't erase old data), which could serve for a new chapter on regression testing if there's a second edition.

Summary of topics covered:

• Terms: unit testing, integration testing, system testing.
• Basics of doctest and unittest, their strengths and weaknesses.
• Using mocks (with Mocker).
• Using Nose.
• Test-Driven Development with lots of example code.
• Using Twill.
• Integration testing with lots of example code.
• Using coverage
• Post-commit hooks to run tests with Bazaar, Mercurial, Git, Darcs, Subversion.
• Continuous integration with Buildbot

I found the TDD cycle a bit larger than I generally like, but I believe it's a matter of taste, and perhaps a shorter cycle wouldn't work as well in a written medium.

I found it a bit jarring how the Twill chapter intrudes between the two chapters showing unit testing and integration testing of the same sample package. I think it would've been better to swap the order of chapters 8 and 9.

I liked the technique presented for picking subsets of the code for integration tests, although I wonder how well it would work on a larger project.

Topics not covered:

• Functional testing (which is very close but not exactly the same as system testing).
• Regression testing (page 46 contains advice about this without mentioning the term regression testing).
• Continuous integration with Hudson (simpler to set up than buildbot, easily covers 80% of cases).

As you can see these holes are all rather small.

Probably the biggest weakness of the book is the complexity of some things shown:

• writing mocks for pure unit tests
• mocking other instances of the same class under test
• even occasionally mocking self, which needs tricks like calling a method's im_func directly
• mocking __reduce_ex__ so you can pickle mocks in an integration test, instead of using real classes or simple stubs.
• testing the same code multiple times: unit tests, several sets of integration tests that test ever-increasing subsets of classes
• Buildbot instead of Hudson

Seeing the repetitive and redundant mock code in the first few doctest examples I started asking what's the point?, but the book failed to provide a compelling answer (the answer provided—it's easier to locate bugs—works just as well for integration tests that focus on individual classes). And there are good answers for that question, like instant feedback from your unit test suite. Are they worth the additional development effort? Maybe that depends on the developer. I don't think they would help me, so I tend to stick with low-level integration tests I call "unit tests" (as well as system tests; it's always a mistake to keep all your tests in a single level). I'm slightly worried that this book might give the wrong impression (testing is hard) and turn away beginning Python programmers from writing tests altogether.

Overall I do not feel that I have wasted my time reading Python Testing. I look forward to reading the other reviews that showed up on Planet Python. I gathered that not all reviewers were happy with the book, but avoided reading their reviews in order not to influence my own.

Profiling a Python program is getting easier and easier:

$ python -m cProfile -o prof.data bin/test -f

runs our test runner (which is a Python script) under the profiler and stores the results in prof.data.

$ runsnake prof.data

launches the RunSnakeRun profile viewer, which displays the results visually:

The square map display of RunSnakeRun, with the 'render_restructured_text' function highlighted.

Who knew that ReStructuredText rendering could be such a time waster? A short caching decorator and the test suite is twice as fast. The whole exercise took me less than an hour. I should've done it sooner.

Other neat tools:

• pstats from the standard library lets you load and display profiler results from the command line (try python -m pstats prof.data).
• pyprof2calltree converts Python profiler data files to a format that the popular profiler visualization tool kcachegrind can understand. It's somewhat less useful now that RunSnakeRun exists.
• profilehooks by yours truly has decorators for easily profiling individual functions instead of entire scripts.
• keas.profile and repoze.profile hook up the profiler as WSGI middleware for easy profiling of web apps.

• Read and review Python Testing: Beginner's Guide and Grok 1.0 Web Development for Packt. (The links are trackable to my blog, but I'm not getting anything out of it. Other than free copies of the e-books, which I already received, in exchange for a promise to review them on this blog.)
• Help Reportlab folks set up continuous integration (most likely Hudson, since Buildbot, while powerful, has a steep learning curve).
• Think about becoming the buildbotmaster for Zope. Originally I intended to volunteer to set up a few buildbots for various Zopeish projects (ZTK, BlueBream, Grok, Zope 2) since half of the existing ones were down or broken. Then various people fixed some of the broken ones and other people chimed in mentioning existing buildbots that nobody else knew about. There is a need for somebody to coordinate all this activity: make sure we have up-to-date test results for all kinds of projects, aggregate them in one place, chase up build slaves for exotic OSes (i.e. Windows)... I don't think I'm well suited for this kind of organisational activity.
• Push along the various scratch-my-itch open source projects (GTimeLog, irclog2html, zodbbrowser).
• No idea what, but I've been wanting to do something for Maemo. Something small, given the copious amounts of free time I have.
• Then there's the paying work. On the plus side, there are opportunities for fun there (today I slashed functional test run time by a half, by adding a small caching decorator in front of a single function. RunSnakeRun and cProfile rule!)
• You know what, scratch the Zope buildbotmaster idea. Maybe I can do something technical there, e.g. a cron script to ping the various buildbot, scrape HTML/parse emails and aggregate build results. Maybe.
• I hope I don't get burnout again. Because that would suck. Again. Been there, done that, didn't even get a T-shirt.

I really ought to read Getting Things Done. Reading it has been on my todo-list for years.

As a penance, here's a Terminal trick for you:

LABELS='[Tab,Esc,Enter,PgUp,PgDn,F2,VKB]'
KEYS='[Tab,Escape,KP_Enter,Page_Up,Page_Down,F2,Return]'
gconftool -s /apps/osso/xterm/key_labels --type list --list-type string "$LABELS"
gconftool -s /apps/osso/xterm/keys --type list --list-type string "$KEYS"

This changes the toolbar to have three extra keys (Enter, F2, and a key that acts like Enter when the hardware keyboard is open, and opens the virtual keyboard if the hardware keyboard is closed).

Update: added screenshot:

Nokia N900 Terminal app with new toolbar buttons

There are IRC logs of the meeting, and Christian Theune posted a summary to the mailing list.

My take on this can be summed up as: Zope ain't dead yet! The project has fragmented a bit (Zope 2, Zope Toolkit, Grok, BlueBream, Repoze), but we all share a set of core packages and we want to keep them healthy.

Next meeting is also happening on a Tuesday, at 15:00 UTC on #zope in FreeNode.

Unfortunately the data from the customers included some \x85 characters, which were breaking the CSV parsing.

0x85 is a control character (NEXT LINE or NEL) in Latin-1, but it's a printable character (HORIZONTAL ELLIPSIS) in Microsoft's code page 1252, which is often mistaken for Latin-1. I would venture a suggestion that the encoding of the customer data was not latin-1 but rather cp1252.

>>> '\x85'.decode('cp1252')
u'\u2026'

Fast-forward to 2008: there are actual users out there (much to my surprise), I still haven't added the originally-envisioned spit and polish, haven't done anything to foster a development community, am wracked by guilt of not doing my maintainerly duties properly, which leads to depression and burnout. So I do the only thing I can think of: run away from the project and basically ignore its existence for a year. Unreviewed patches accumulate in my inbox.

It seems that the sabbatical helped: yesterday, triggered by a new Debian bug report, I sat down, fixed the bug, implemented a feature, applied a couple of patches languishing in the bug tracker, and released version 0.3 (which was totally broken thanks to setuptools magic that suddenly stopped working; so released 0.3.1 just now). Then went through my old unread email, created bugs in Launchpad and sent replies to everyone. Except Pierre-Luc Beaudoin, since his @collabora.co.uk email address bounced. If anyone knows how to contact him, I'd appreciate a note.

There are also some older changes that I made before I emerged out of the funk and so hadn't widely announced:

• There's a mailing list for user and developer discussions (if there still are any ;).
• GTimeLog's source code now lives on Launchpad (actually, I mentioned this on my blog once).

The current working directory is my project configuration/state. I run tests here (bin/test), I search for code here (vim -t TagName, find + grep), I run applications here (make run or bin/appname). I can multitask freely, for example, if I'm in the middle of typing an SVN commit message, I can hit Ctrl+Shit+T, get a new terminal tab in the same working directory, and look something up. No aliases/environment variables/symlinks/scripts making changes to config files. I can work on multiple projects at the same time. I can work remotely (over ssh).

Gary Bernhardt's screencasts on Vimeo show how productive you can get if you learn Vim and tailor it to your needs. I have Vim scripts that let me

• See the name of the class and function that I'm editing in the statusbar, even if the class/function definition is offscreen: pythonhelper.vim.
• See all pyflakes warnings and errors in a list as soon as I press F2 to save the file: python_check_syntax.vim.
• Add a "from foo.bar import Something" line at the top of the file if I press F5 when my cursor is on Something, looking up the package and module from ctags: python-imports.vim.
• Switch between production code and unit tests with a single key if the project uses one of several conventions for tests (e.g. ./foo.py <-> ./tests/test_foo.py): py-test-switcher.vim.
• Generate a command line for running one particular unit test (the one my cursor is inside) and copy it into the system clipboard, so I can run that test by Alt-Tabbing into my terminal window and pasting. py-test-runner.vim.
• Open the right file and move the cursor to the right line if I triple-click a line of traceback in a shell (or an email) then press F7 in my gvim window: py-test-locator.vim.
• Compare my version of the code with the pristine version in source control in an interactive side-by-side diff that lets me revert bits I no longer want: vcscommand.vim.
• Highlight which lines of the source are covered by my tests, if I have coverage information in trace.py format: py-coverage-highlight.vim.
• Show the signature of a function/class's __init__ when I type the name of that class/function and an open parenthesis (looked up from tags): py-function-signature.vim.
• Fold code into an outline so I only see names of methods or classes instead of their full bodies: vimrc, function PythonFoldLevel.
• Fold diff files so I can see whole hunks/files and can delete those with a single key (well, two keys -- dd). Useful for reviewing large diffs (tens of thousands of lines): vimrc, function DiffFoldLevel.

Some of these come from www.vim.org, some I've written myself, some I've taken and modified a little bit to avoid an irritating quirk or add a missing feature. Some things I don't have (and envy Emacs or IDE users for having -- like an integrated debugger for Python apps, and, generally, integration with other tools, running in the background).

It's been my plan for a long time to polish my plugins, release them somewhere (github? bitbucket? launchpad?) and upload to vim.org, but as it doesn't seem to be happening, I thought I'd at least put an svn export of my ~/.vim on the web.

Here's a pattern I've been using in Zope 3 to display multiline user-entered text as several paragraphs:

import cgi

from zope.component import adapts
from zope.publisher.browser import BrowserView
from zope.publisher.interfaces import IRequest


class SplitToParagraphsView(BrowserView):
    """Splits a string into paragraphs via newlines."""

    adapts(None, IRequest)

    def paragraphs(self):
        if self.context is None:
            return []
        return filter(None, [s.strip() for s in self.context.splitlines()])

    def __call__(self):
        return "".join('<p>%s</p>\n' % cgi.escape(p)
                        for p in self.paragraphs())

View registration

<configure
    xmlns="http://namespaces.zope.org/zope">

  <view
      for="*"
      name="paragraphs"
      type="zope.publisher.interfaces.browser.IBrowserRequest"
      factory=".views.SplitToParagraphsView"
      permission="zope.Public"
      />

</configure>

and usage

<p tal:replace="structure object/attribute/@@paragraphs" />

Update: The view really ought to be registered twice: once for basestring and once for NoneType. I was too lazy to figure out the dotted names for those (or check if zope.interface has external interface declarations for them), so I registered it for "*". You should know that this makes the view available for arbitrary objects (but won't work for most of them, since they don't have a splitlines method), and that it is, sadly, accessible to users who may try to hack your system by typing things like @@paragraphs in the browser's address bar. Ignas Mikalajūnas offers an alternative solution using TALES path adapters.

Nokia surprised everyone on the first day by handing out 300 pre-release N900s to the participants. I'm so happy now that after a long period of wavering I finally decided to come to the summit! The device is much better than I expected/feared (and I haven't even put a SIM card in yet). We're supposed to provide feedback and will have to send the devices back to Nokia in 6 months. (Nokia insisted on loan contracts signed in blood, kidding, but there are contracts.)

The tiny pixels are beautiful. It's what, 266 pixels per inch? Even older 225 dpi devices spoiled me: both the first generation iPhone and the first generation Kindle displays seemed very coarse and pixellated.

The user interface is very smooth. Having a composition manager improves apparent responsiveness: even if the app is swapped out and not ready to redraw, switching between windows appears to be instant since the picture is cached. And there's no flicker while the apps are redrawing. (Flickering during redraw is one of the main reasons I did not buy a S60 phone and stayed with good old S40.) Speaking of swapping, it's barely noticeable. You can run more apps than fit in RAM without having to suffer. The flash memory is noticeably faster than in a N810. And there's more of it (32 gigs: 28 gig partition for user data, the rest for the system: swap, applications, config files, etc.)

The design and usability of the user interface have improved a lot since the N810. The UI is pretty. Many of the apps are now convenient to use. Pervasive kinetic scrolling is sweet (except when you have really long lists or web pages, then it takes forever to reach the end).

Finally there are PIM-y things people missed in older Maemo releases: calendaring, contacts that can record all kinds of information (such as phone numbers).

All right, enough gushing. There were some irritating things too. For example, Bluetooth support is buggy/incomplete in the pre-release firmware, so it's hard to transfer files. Calendar/contacts sync with S40 phones does not work either. GPS is utterly useless when you're offline (no maps, or at least I haven't found a way to pre-download and cache them; also very long fix times without network assistance). Since I have no desire to pay extortionist roaming charges of my provider (2.5 EUR per megabyte), and haven't had a chance to go look for a prepaid SIM card, I usually have either WiFi or GPS coverage, but not both.

As you can guess, playing the device diverted a part of my attention from the presentations somewhat. I tried to compensate for that by reporting on the talks on IRC (using xchat on the device). I think the strategy backfired; IRC is rather disruptive and the channel is quite busy lately.

You need a web server running Apache and SSH. Enable mod_proxy and mod_proxy_connect and add this to the first (i.e. default) virtual host configuration:

<VirtualHost whatever:80>
...

  # allow ssh to localhost over http proxy
  ProxyRequests on
  AllowCONNECT 22
  <Proxy localhost>
    Order allow,deny
    Allow from all
  </Proxy>

</VirtualHost>

On the client side you need proxytunnel. Sadly, it's not packaged for Ubuntu yet, but compiling from sources is trivial. Edit ~/.ssh/config and add an entry for your proxied ssh connection:

Host pmyservername
ProxyCommand proxytunnel -q -p myserver.mydomain.com:80 -d localhost:22

That's it. Now you can ssh pmyservername. (The p prefix is a reminder that I'm using a proxied connection: ssh fridge versus ssh pfridge. Also it reminds me of Terry Pratchett's Pyramids.).

For extra fun (e.g. IRC) use ssh's built-in SOCKS5 proxy: ssh -D 1080 pmyservername. Then tell the apps to use a SOCKS5 proxy on localhost. Since telling each app to use a proxy (and then, later, telling it to stop using it) is a big *pain*, and some apps (e.g. ssh) don't support proxies directly, a wrapper like tsocks is handy. Edit /etc/tsocks.conf and set the default socks server to 127.0.0.1, then use it to run apps:

$ tsocks xchat-gnome
$ tsocks bzr push lp:myprojectname

tsocks is packaged for Ubuntu.

If your hotel doesn't have free WiFi, a prepaid SIM card with 3G access could be cheaper than roaming charges. Apparently you can get one with a virtually unlimited (for a short stay, anyway) data plan for 27 EUR in Amsterdam.

I want a system

• that is transparent to the user: if my database is at schema version 1, and my code is at version 3, I want it to be automatically upgraded to version 3 on server startup.
• that is not too hard on the programmer: dropping a numbered Python or SQL script in a directory ought to be sufficient to define a transition from schema version X to schema version X+1.
• that handles errors gracefully: makes a backup of the database with the old schema version; runs my script in a transaction and aborts that transaction if the conversion fails (while showing me enough information to debug the problem).
• allows prototyping without having to increment the schema number for every little change I make to the models; I should be the one who decides that a new schema is ready to go out to the world.

I've been glancing at SQLAlchemy-Migrate, since I've been brought up to believe NIHing is Bad. But Migrate is scary. I have to admit that the longer I stare at its documentation, the less I can describe why I think so. All those shell commands—but there's an API for invoking them from Python, so maybe I can achieve my goals. I'll have to try and see.

I will omit bzr commits from this narrative as it's getting long; you can assume that every self-contained change was committed separately.

tests

First, I want a bin/test script to run the test suite. Pylons uses nose, so we need to tell buildout to install the nosetests script (under a different name, since I'm used to typing bin/test no matter what test runner a project happens to use):

$ bzr diff
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-15 19:49:11 +0000
+++ buildout.cfg	2009-09-15 19:49:18 +0000
@@ -8,5 +8,8 @@
 recipe = zc.recipe.egg
 eggs = Pylons
        PasteScript
+       nose
        asharing
 interpreter = python
+scripts = paster
+          nosetests=test

$ bin/buildout
...
Generated script '/tmp/AlliterationSharing/bin/paster'.
Generated script '/tmp/AlliterationSharing/bin/test'.
...
$ bin/test

----------------------------------------------------------------------
Ran 0 tests in 0.276s

OK

ctags

Documentation is good, but sometimes you want to look at the source code of the framework. There's a tool called ctags that builds a database of identifiers. The popular text editors Vim and Emacs can then use the tags database to jump to a definition of any name with a single keystroke (Ctrl-] in vim, M-. in emacs).

Building the tags database is complicated by each Python package being installed into a separate directory. There's a buildout recipe called z3c.recipe.tag that finds those directories and lets you build a unified tags file. We'll also ask buildout to make sure it unzips any packages distributed as .egg files, since ctags doesn't process those:

$ bzr diff
@@ -1,8 +1,9 @@
 [buildout]
 develop = .
-parts = pylons
+parts = pylons ctags
 
 newest = false
+unzip = true
 
 [pylons]
 recipe = zc.recipe.egg
@@ -13,3 +14,7 @@
 interpreter = python
 scripts = paster
           nosetests=test
+
+[ctags]
+recipe = z3c.recipe.tag:tags
+eggs = ${pylons:eggs}

$ bin/buildout
...
Generated script '/tmp/AlliterationSharing/bin/ctags'.
...
$ bin/ctags

omelette

ctags lets you find classes and functions by name; it doesn't let you find packages or modules. There's another recipe, collective.recipe.omelette that creates a tree of symlinks mirroring the Python package structure (here 'unzip = true' also comes in handy):

$ bzr diff
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-15 20:04:42 +0000
+++ buildout.cfg	2009-09-15 20:05:30 +0000
@@ -1,6 +1,6 @@
 [buildout]
 develop = .
-parts = pylons ctags
+parts = pylons ctags omelette
 
 newest = false
 unzip = true
@@ -18,3 +18,7 @@
 [ctags]
 recipe = z3c.recipe.tag:tags
 eggs = ${pylons:eggs}
+
+[omelette]
+recipe = collective.recipe.omelette
+eggs = ${pylons:eggs}

$ bin/buildout 
...
$ ls -l parts/omelette
...

The symlink tree is created under parts/omelette/. For example, if you want to see what webhelper tags were available, you can open parts/omelette/webhelper/html/builder.py in your editor and see.

Makefile

This is getting long (and not everyone may be interested¹), but one long post is easier to skip than five medium ones in a row, so I'll continue.

¹ Sorry, Planet Maemo! There's an RSS feed of posts tagged 'maemo', if you can figure out the URL, which is very well hidden by PyBlosxom, *sigh*.

Wouldn't it be nice if new developers could check out your project and start it up with just a couple of commands? Make is a time-tested tool that works well for this:

$ cat Makefile
# Just remember that you need to use real tabs, not spaces, in a Makefile

PYTHON = python

.PHONY: all
all: bin/paster

.PHONY: run
run: bin/paster
        bin/paster serve development.ini --reload

.PHONY: test check
test check: bin/test
        bin/test

.PHONY: tags
tags: bin/ctags
        bin/ctags

bin/paster bin/test bin/python bin/ctags: bin/buildout
        bin/buildout

bin/buildout: bootstrap.py
        $(PYTHON) bootstrap.py

Now all you need to do after checking out is run 'make' to set up a working development environment. 'make run' or 'make test' will also do that, if necessary, so this one-liner is sufficient to get a working Hello World application on port 5000:

$ bzr branch lp:~mgedmin/+junk/AlliterationSharing && cd AlliterationSharing && make run

Try it! You'll get a Bazaar branch with all the history of this little blog project.

To start a new Pylons project, first create an empty directory. Let's call our new project AlliterationSharing¹, because everybody is sick of 'foo' and 'bar'.

¹ Generated by randomly picking two words from /usr/share/dict/words, then chosen over among 120 other variants that weren't as good.

$ mkdir -p ~/src/AlliterationSharing
$ cd ~/src/AlliterationSharing

Now create a file called buildout.cfg with the following content:

$ cat buildout.cfg
[buildout]
parts = pylons

[pylons]
recipe = zc.recipe.egg
eggs = Pylons
       PasteScript
interpreter = python

Download bootstrap.py to it and run it to get bin/buildout. Note: you can chose which Python version you want to use by running bootstrap.py with it. All other scripts under bin/ will be generated by buildout and will use the same Python interpreter.

$ wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
$ python bootstrap.py
Creating directory '.../AlliterationSharing/bin'.
Creating directory '.../AlliterationSharing/parts'.
Creating directory '.../AlliterationSharing/eggs'.
Creating directory '.../AlliterationSharing/develop-eggs'.
Generated script '.../AlliterationSharing/bin/buildout'.

Run bin/buildout to install Pylons into your sandbox.

$ bin/buildout
Installing pylons.
Generated script '.../AlliterationSharing/bin/paster'.
Generated interpreter '.../AlliterationSharing/bin/python'.

Aside: buildout has this very nice feature where it can share Python packages between projects. This will save you enormous amounts of time that would otherwise be spent downloading and unpacking eggs. To make use of this facility, create a file ~/.buildout/default.cfg with

$ cat ~/.buildout/default.cfg 
[buildout]
eggs-directory = /home/mg/tmp/buildout-eggs
# XXX replace /home/mg with the full path of *your* home directory
# it would be much nicer if buildout let me use ~ or $HOME
# see https://bugs.launchpad.net/zc.buildout/+bug/190260

Another useful trick is to pass the -N flag to bin/buildout, which will tell it not to bother looking for newer versions of packages on the Internet when there's already an existing version installed in your eggs directory.

Back to business: now you've got two new scripts: bin/python and bin/paster. You can use the first one to play with the interactive Python console where you can now import pylons and all the dependencies; it has no other value.

Now is a good point to add the files you've created into a version control system. I'll arbitrarily use Bazaar.

$ bzr init .
$ bzr add bootstrap.py buildout.cfg
$ bzr ignore bin parts eggs develop-eggs .installed.cfg
$ bzr commit -m "Create AlliterationSharing project"

Run bin/paster create -t pylons to create a skeleton project.

$ bin/paster create -t pylons asharing
$ bzr ignore *.egg-info
$ bzr add asharing
$ bzr commit -m "Generated project files with paster create"

Now paster creates a directory structure that I don't like:

AlliterationSharing/
  buildout.cfg
  bin/
  asharing/
    setup.py
    README.txt
    MANIFEST.in
    asharing/
      __init__.py
      config/
      controllers/
      templates/
      public/

I'd like the README and setup.py to be in the top level, and I dislike repeating 'asharing' twice in directory names. I'll move some files around

$ cd asharing/
$ bzr mv development.ini docs MANIFEST.in README.txt setup.* test.ini ../
$ bzr rm ez_setup.*
$ cd ..
$ bzr mv asharing src
$ bzr ci -m "Moved some files around"

Now the tree looks like this:

AlliterationSharing/
  buildout.cfg
  setup.py
  README.txt
  MANIFEST.in
  bin/
  src/
    asharing/
      __init__.py
      config/
      controllers/
      templates/
      public/

We have to tell setup.py where to find the source tree

$ bzr diff
=== modified file 'MANIFEST.in'
--- MANIFEST.in	2009-09-13 13:04:00 +0000
+++ MANIFEST.in	2009-09-13 13:05:59 +0000
@@ -1,3 +1,3 @@
-include asharing/config/deployment.ini_tmpl
-recursive-include asharing/public *
-recursive-include asharing/templates *
+include src/asharing/config/deployment.ini_tmpl
+recursive-include src/asharing/public *
+recursive-include src/asharing/templates *

=== modified file 'setup.py'
--- setup.py	2009-09-13 13:04:00 +0000
+++ setup.py	2009-09-13 13:04:40 +0000
@@ -17,7 +17,8 @@
         "SQLAlchemy>=0.5",
     ],
     setup_requires=["PasteScript>=1.6.3"],
-    packages=find_packages(exclude=['ez_setup']),
+    packages=find_packages('src', exclude=['ez_setup']),
+    package_dir={'': 'src'},
     include_package_data=True,
     test_suite='nose.collector',
     package_data={'asharing': ['i18n/*/LC_MESSAGES/*.mo']},

(I'm not sure if you also need to change package_data and/or setup.cfg; it's possible that I left i18n in a broken state. Can somebody comment on this?)

And we have to tell buildout that we've got a new Python package to enable in the project environment

$ bzr diff buildout.cfg 
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-13 12:57:21 +0000
+++ buildout.cfg	2009-09-13 13:08:05 +0000
@@ -1,8 +1,10 @@
 [buildout]
+develop = .
 parts = pylons
 
 [pylons]
 recipe = zc.recipe.egg
 eggs = Pylons
        PasteScript
+       asharing
 interpreter = python

Now you can re-run bin/buildout and start your hello-world project

$ bzr commit -m "Include the new package in the build"
$ bin/buildout -N
$ bin/paster serve --reload development.ini

Happy hacking!

To be continued: telling buildbot to create bin/test; using ctags and omelette.

(Recorded with byzanz. My gif-fu is nonexistent or I would make it loop, but with a sufficiently long delay at the end to avoid irritation. Now you have to reload the whole page if you missed the animation.)

I'm somewhat ambivalent about the animation effect. On one hand, shiny! On the other hand, hitting tiny clickable areas is not good usability. Still, shiny!

Footnotes are kind of a personal pet-peeve of mine.

svn co svn+ssh://svn.zope.org/repos/main/plone.z3cform/trunk plone.z3cform
cd plone.z3cform
python2.4 bootstrap.py
bin/buildout
bin/test -pvc

Now suppose you want to change the buildout environment somehow, e.g. use the current development version of zope.testing instead of whatever is specified in buildout.cfg. Don't edit the existing buildout.cfg (you might accidentally commit your local debug changes), instead create a new cfg file, e.g. test.cfg:

[buildout]
extends = buildout.cfg
develop += ../zope.testing

[versions]
# override any existing version pins
zope.testing =

Now re-run buildout

bin/buildout -c test.cfg
bin/test -pvc

And the tests should be run with the newest zope.testing.code.

Only this does not work with plone.z3cform, and I have no clue why. It generally works with other packages (at least those that use the zc.recipe.testrunner rather than collective.recipe.z2testrunner). Buildout is like that sometimes :(

Updated gtkeggdeps, the interactive Python package dependency browser. Collaborated with Thomas Lotze, who maintains the engine (tl.eggdeps) that gtkeggdeps wraps, to resolve API mismatches. Moved the sources to launchpad.net, added a test suite, made it use zc.buildout for convenient development.

Moved the source repository of gtimelog, the simple desktop time tracker, to launchpad.net. Failed to do anything else with it. :-(

Tried to work on xdot, wrestled with git-svn merges, failed abysmally. Asked upstream to upload xdot to PyPI.

Released ZODB Browser, but this deserves a separate post.

Sent a bunch of pyflakes patches from my old branch upstream, created trac tickets for the rest. Wrestled with bzr-svn merges, failed abysmally.

Question: are there any df-like utilities that filter out all the crap and show only interesting bits? The standard df as well as pydf both display 8 lines instead of 1. Discus is worse: it shows 20. GUI utilities like Baobab also suffer from this confusion, especially bind mounts.

Ironically, Ubuntu's update-motd gets confused by Ubuntu's private user directories and displays disk stats for ~/Private as if it were a real partition.

The question, slightly paraphrased: given a class

class LameContainerOld:
    def __init__(self):
        self._items = {'bar':'test'}
 
    def __getitem__(self, name):
        return self._items[name]
 
    def __getattr__(self, attr):
        return getattr(self._items, attr)

why does the 'in' operator work

>>> container = LameContainerOld()
>>> 'foo' in container
False
>>> 'bar' in container
True

when the equivalent new-style class raises a KeyError: 0 exception? Also, why does __getattr__ appear to be called to get the bound __getitem__ method of the dict?

>>> container.__getitem__
<bound method LameContainerNew.__getitem__ of {'bar': 'test'}>

What actually happens here is that LameOldContainer.__getattr__ gets called for special methods such as __contains__ and __repr__. This is why (1) the 'in' check works, and (2) it appears, at first glance, that you get the wrong __getitem__ bound method. If you pay close attention to the output, you'll see that it's the __getitem__ of LameOldContainer; it's just that repr(LameOldContainer()) gets proxied through to the dict.__repr__ when you don't expect it:

>>> container
{'bar': 'test'}

First: don't be scared by the list of comment-related plugins on the PyBlosxom site. There's only one important plugin: comments. All others depend on it and enhance its functionality. The last three or four times I was about to add comments to my blog I got scared at step one: evaluate the available plugins. Don't repeat my mistake!

Second, follow the instructions carefully. There's no shortcut.

Third, fix what's broken. Be prepared to debug the source code. print >> sys.stderr, "message" is your friend.

Fourth, fiddle with the look (CSS and HTML).

Fifth, write a blog post and eagerly await your first comments.

Step 3 screams for an explanation, doesn't it? Problem 1: the comments plugin requires that you use categories in your blog. I'm not (I'm holding out for tags). Workaround: comment out if entry['absolute_path'] check in cb_story and cb_story_end.

Problem 2: the AJAX post returns "Empty response from server". Workaround: modify cb_story_end to call readComments directly if entry['num_comments'] is None, since cb_story, which usually does the read, is not called during the AJAX post.

Problem 3: if you enable comment moderation (by setting comment_draft_ext to a different value from comment_ext), the AJAX post returns "Empty response from server" once more. Workaround: modify cb_prepare to notice this case and set data['moderated'] = True, create a new template comment-moderated and render it in cb_story_end just like the preview template is rendered; also modify __shouldOutput to return True when rendering comment-moderated.

I'll post patches to the pyblosxom mailing tomorrow, unless I forget. It's 6 am already, and I'm kind of sleepy. I just hope I haven't inadvertently broken my RSS feed or flooded any planets.

Oh, and a helpful hint: don't name the post you're writing comments.txt, or the #comments anchor will point to the start of the story instead of the comments.