Consider this: we have two loggers

root
  \
   -- mylogger

configured as follows:

import logging
root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(logging.FileHandler("info.log"))
mylogger = logging.getLogger('mylogger')
mylogger.setLevel(logging.DEBUG)
mylogger.addHandler(logging.FileHandler("debug.log"))

What happens when I do mylogger.debug('Hi')?

Answer: the message appears in both debug.log and info.log.

That was surprising to me. I'd always thought that a logger's level was a gatekeeper to all of that logger's handlers. In other words, I always thought that when a message was propagating from a logger to its parent (here from mylogger to root), it was also being filtered against the parent's log level. That turns out not to be the case.

What actually happens is that a message is tested against the level of the logger where it was initially logged, and if it passes the check, it gets passed to all the handlers of that logger and all its ancestors with no further checks. Unless the propagation is stopped somewhere by one of the loggers having propagate set to False. And of course each handler has its own level filtering. And I'm ignoring filters and the global level override.

Part of the confusion was caused by my misunderstanding of logging.NOTSET. I assumed, incorrectly, that it was just a regular logging level, one even less severe than DEBUG. So when I wrote code like this:

import logging
root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(logging.FileHandler("info.log"))
mylogger = logging.getLogger('mylogger')
mylogger.setLevel(logging.NOTSET)  # <-- that's the default level, actually
mylogger.debug("debug message")

I saw the debug message being suppressed and assumed it was because of the root logger's level. Which is correct, in a way, just not the way I thought about it.

NOTSET does not mean "pass all messages through", it means "inherit the log level from the parent logger". The documentation actually describes this, although in a rather convoluted way. My own fault for misunderstanding it, I guess.

import subprocess
with open('hi.txt', 'w') as f:
  subprocess.Popen(['echo', 'Hello'], stdout=f).wait()

Here's what it does:

$ python2.7 hi.py
$ cat hi.txt
Hello

And here's what happens if you try to use the trace module to trace the execution:

$ python2.7 -m trace --trace hi.py
(lots of trace output omitted)

So far so good, but take a look at hi.txt:

$ cat hi.txt
subprocess.py(1170):                             _dup2(errwrite, 2)
 --- modulename: subprocess, funcname: _dup2
subprocess.py(1164):                                 if a == b:
subprocess.py(1166):                                 elif a is not None:
... 154 lines of unexpected trace output omitted ...
os.py(379):         try:
os.py(380):             func(fullname, *argrest)
Hello

The tracing hook installed by trace.py survives the os.fork() and interferes with subprocess output redirection. Now you know.

Imagine what this does to scripts that execute shell commands and try to parse their output.

Who wants to file a bug and/or provide a patch? I'm afraid I don't have enough round tuits...

By default fileConfig disables all pre-existing loggers if they (or their parent loggers) are not explicitly mentioned in your .ini file.

This can result in unintuitive behaviour:

(if you don't see the embedded example, you can find it at https://gist.github.com/1642893).

If you have Python 2.6 or later (and you should), you can turn this off by passing disable_existing_loggers=False to fileConfig(). But what if it's not you calling fileConfig() but your framework (e.g. the above-mentioned paster serve)?

Now usually paster serve tries to configure logging before importing any of your application modules, so there should be no pre-existing loggers to disable. Sometimes, however, this doesn't work for one reason or another, and you end up with your production server suppressing warnings and errors that should not be suppressed. I haven't actually figured out yet who's responsible for those early imports in the application I'm working on (until today I assumed, incorrectly, that paster imports the module containing your WSGI app before it calls fileConfig).

If you're not sure if this bug can bite you or not, check that you don't have any disabled loggers by doing something like

import logging
assert not any(getattr(logger, 'disabled', False)
               for logger in logging.getLogger().manager.loggerDict.values())

while your application is running.

The rest of this post will describe the conversion (and verification) in detail, because if I ever need to do this again, I do not want to start from scratch.

Part one: unexpected gift

Raffaele Salmaso did a heroic job and sent me a tarball with a mercurial repository, produced with "something like" this:

> hg clone --layout single $SVN repo-tmp
> hg convert --filemap filemap repo-tmp repo
> cd repo
> hg qinit
> hg qimport -r 0:tip
> hg qpop -a
> cd .hg/patches
> check patches for correctness
> fix tags (svn are different from mercurial ones)
> hg qfinish -a

The conversion had only two problems:

1. it introduced new commits that modify a new file .htags
2. it changed the contents of README.txt in changeset 53 (corresponding to svn revision 55, "Allow branch names to have prefixes.") and newer versions:

--- svn version
+++ hg version
-  *scheme://server/path/to/svn/repo*/*subdirs*
+  *scheme://server/path/to/svn/repo*/trunk/*subdirs*

I did not notice either problem at first.

Part two: conversion to git

Note: I wrote this up after I've done everything up to and including part five. The commands and directory names here are not the actual commands and directories I've used; although I tried to be accurate. I've also skipped some false trails.

Converting hg to git was pretty easy:

$ mkdir -p /tmp/conv
$ cd /tmp/conv
$ tar xvjf eazysvn_20120220-133823.tar.bz2
creates /tmp/conv/eazysvn/
$ mkdir eazysvn-git
$ cd eazysvn-git
$ git init
$ hg-fast-export -r /tmp/conv/eazysvn
converts, leaves no working tree; git status shows all files as deleted
$ git checkout
restore working tree

I was a bit surprised by git status showing a bunch of deleted files at the end there. I suppose hg-fast-export expects to be run inside a bare repository, or maybe it expects the user to know enough git to understand what happened and do the git checkout if necessary.

Part three: cleanup

I wanted to drop the empty changesets that were introduced by Raffaele's conversion process for modifying .hgtags. Since the manual page for git filter-branch had an example for dropping all empty changesets, I used it directly.

$ git filter-branch --commit-filter 'git_commit_non_empty_tree "$@"'

This also dropped some changesets that were present in my Subversion repository -- those that manipulated svn properties, and the one that moved everything in svn root under /trunk. I won't miss those.

Note: if you try

$ git filter-branch --commit-filter='git_commit_non_empty_tree "$@"'

(i.e. '=' instead of a space after --commit-filter), you will get a completely baffling error message that doesn't even hint at what is wrong.

At this point I ran gitk --all to look around and discovered that git filter-branch left all the tags pointing to obsolete revisions. I created new tags manually with gitk, including some that were missing in my svn repository. Every release since 1.6.0 is now tagged (releases before that did not have source tarballs on PyPI, so I had no way to verify which checkin corresponded to which release). I also changed the tag naming scheme to be "v1.x.y" instead of just "1.x.y".

Oh, and to get rid of the old history I did

$ git tag -d 1.9.0 1.11.0 1.12.0 1.12.1
$ git gc --prune

Part four: verification

I downloaded all the available releases from PyPI:

$ mkdir -p /tmp/verify
$ cd /tmp/verify
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   wget http://pypi.python.org/packages/source/e/eazysvn/eazysvn-$v.tar.gz
>   tar xvzf eazysvn-$v.tar.gz
> done

Exported all of my git tags:

$ mkdir -p /tmp/verify/git
$ cd /tmp/conv/eazysvn-git
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   git archive --format=tar --prefix eazysvn-$v/ v$v | gzip \
>       > /tmp/verify/git/eazysvn-$v-git.tar.gz
> done
$ cd /tmp/verify/git
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   tar xvzf eazysvn-$v-git.tar.gz
> done

And compared them:

$ diff -ur /tmp/verify /tmp/verify/git

I expected to see "Only in dir1: setup.cfg" messages only for things like 'eazysvn.egg-info' or 'PKG-INFO'. Unfortunately this is where actual differences I mentioned in part one showed up: in README.txt for all trees starting with release 1.9.0.

Part five: rectification

I needed to rewrite history again, but I didn't want to use git filter-branch this time (I didn't want to manually tag all the releases again). I tried fast-export:

$ cd /tmp/conv/eazysvn-git
$ git fast-export --all > ../EXPORT.txt
$ vim ../EXPORT.txt
search and replace 'repo*/*subdirs*' with 'repo*/trunk/*subdirs*'
fix up the file size above each change (increment by 6, the length of 'trunk/')
$ mkdir /tmp/conv/eazysvn-git2
$ cd /tmp/conv/eazysvn-git2
$ git init
$ git fast-import < ../EXPORT.txt
succeeds, leaves no working tree; git status shows all files as deleted
$ git checkout
restore working tree
$ gitk --all

Everything looked about right.

Part six: final verification

But was it actually right?

$ cd /tmp/verify/
$ mkdir pypi
$ mv eazysvn-*/ pypi/
$ rm -rf git/*
$ cd /tmp/conv/eazysvn-git2
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   git archive --format=tar --prefix eazysvn-$v/ v$v \
>       | (cd /tmp/verify/git && tar -xf - )
> done
$ cd /tmp/verify
$ diff -ur pypi git

Yes!

Part seven: uploading to GitHub

It was all plain sailing from here:

$ cd /tmp/conv/eazysvn-git2
$ git remote add origin git@github.com:mgedmin/eazysvn.git
$ git push -u origin master
$ git push --tags

And then there were documentation updates (to point to GitHub instead of the old Subversion repository), Makefile updates (make release makes sure my sdist contains everything I have in my repository, because I've been bitten by setuptools magic before), etc.

I also released eazysvn 1.12.2 to PyPI to test my Makefile changes, and because there were unreleased changes that should've been released a long time ago.

So that's it. Only took me three hours from the point where I found a Mercurial repository in my inbox.

git-svn is unsuitable for the conversion, because in revision 50 I moved / to /trunk and added the traditional /tags and /branches. With git-svn I either get one third of the history that ignores everything before the layout switch, or I get directories named 'trunk', 'tags' and 'branches'.

Then I thought maybe hg would be smarter about the conversion, and then I could use hg-fast-export to convert hg to git. I enabled the hgsubversion extension:

$ sudo apt-get install hgsubversion
$ echo '[extensions'] >> ~/.hgrc
$ echo 'hgsubversion =' >> ~/.hgrc

(blog posts like this are a good reason why hg ought to steal the 'git config --global foo.bar=baz' syntax from git).

Then I converted the svn repository to Mercurial:

$ hg clone svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-hg

hg log -p confirmed that hg handled the conversion nicely looked right-ish at first glance, except the author information was nonsensical. To fix that:

$ rm -rf eazysvn-hg
$ echo 'mg = Marius Gedminas <marius@gedmin.as>' > AUTHORS
$ hg clone svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-hg -A AUTHORS

Unfortunately, a closer look at hg log now shows that two thirds of the history is lost: hg ignored everything before the layout restructuring, despite printing the log messages of those revisions as it went about the conversion. *sigh*.

Dear lazyweb, surely svn layout reorganization can't be such a rare thing that no tools in existence support it? What should I try next?

P.S. I also tried Bazaar, to see what it would do:

$ bzr branch svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-bzr
Repository with UUID 4fc293c4-4eed-0310-a01a-b4ad72f90fad at svn+ssh://fridge/home/mg/svn/eazysvn contains fewer revisions than cache. This either means that this repository contains an out of date mirror of another repository (harmless), or that the UUID is being used for two different Subversion repositories (potential repository corruption).
bzr: ERROR: exceptions.KeyError: 'missing revision paths for 78'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 946, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1150, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 699, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 721, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1263, in run
    from_location)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 919, in open_tree_or_branch
    return bzrdir._get_tree_branch()
  File "/usr/lib/python2.7/dist-packages/bzrlib/controldir.py", line 410, in _get_tree_branch
    branch = self.open_branch(name=name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/remote.py", line 420, in open_branch
    branch_path = self._determine_relpath(name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/remote.py", line 369, in _determine_relpath
    layout = repos.get_layout()
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 701, in get_layout
    return self.get_layout_source()[0]
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 720, in get_layout_source
    self._find_guessed_layout(self.get_config())
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 743, in _find_guessed_layout
    revnum, self._hinted_branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 143, in repository_guess_layout
    return logwalker_guess_layout(repository._log, revnum, branch_path=branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 149, in logwalker_guess_layout
    logwalker.iter_changes(None, revnum, max(0, revnum-GUESS_SAMPLE_SIZE)), revnum, branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 104, in guess_layout_from_history
    for (revpaths, revnum, revprops) in changed_paths:
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/logwalker.py", line 60, in iter_all_changes
    revpaths = get_revision_paths(revnum)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/logwalker.py", line 295, in get_revision_paths
    return self.cache.get_revision_paths(revnum)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/cache/tdbcache.py", line 187, in get_revision_paths
    raise KeyError("missing revision paths for %d" % revnum)
KeyError: 'missing revision paths for 78'

You can report this problem to Bazaar's developers by running
    apport-bug /var/crash/bzr.1000.2012-02-19T22:12.crash
if a bug-reporting window does not automatically appear.

Vienna: great transportation system, delicious food (either that, or I was always extremely hungry when I ate), huge portions, restaurants open until midnight. Shame I didn't have time to see the city itself.

The N9 is a gorgeous phone; much more so in real life than in pictures.

After some hassle upgrading the Qt SDK (the provided upgrade tool managed to somehow remove the Qt Creator IDE while purporting to upgrade it; I had to reinstall the entire SDK) and flashing my N950 with Beta 2 firmware (Qt SDK 1.1.3 produces apps incompatible with the old Beta 1 firmware of the N950) I started prototyping a time tracking application in QML, while learning QML and the N9 design guidelines at the same time.

Converting the pretty pictures into QML was harder than I expected, but at the end of the second day I had something that looked like a native N9 application.

Most useful reference pages were:

• The QML syntax introduction (which felt incomplete, but was almost adequate in the end).
• The list of Harmattan-specific QML components.
• The list of standard QML components.
• The UI building blocks pages mentioned above (pretty pictures! pretty colours! I like shiny things!).
• Harmattan Qt Components User Interface Guidelines: pixel and font sizes of the standard UI elements. (Ignore the "import UIConstants.js" red herrings; it's an internal thing apparently, and you can't use it directly from your 3rd-party apps. Unless you find and copy UIConstants.js into your project, after figuring out if the licence allows it, which seemed too much of a hassle for me to even start. So I hardcoded all the numbers directly, like a bad programmer who doesn't know about constants.)
• The TutorialApplication sample, finding the sources of which was unexpectedly difficult -- a straight git clone of the qt-components repository gives you something too recent to run with the older qt-components version on the N9. I ended up using apt-get source qt-components in Scratchbox to download the source tarball of version 1.0~git20110525-1+0m6. Look in qt-components/examples/meego/.

Finally, workflow. QML is parsed at run time (application startup time, unless you delay the loading), which means no recompilation ought to be required to make changes, which means short development feedback cycles ought to be possible. So I was not happy about having to wait several seconds after hitting Run in Qt Creator, while it built a .deb, copied it to the device over wifi or USB networking, installed and ran it there. Deploying to the Qt Simulator is quicker, but not as much as I think it ought to be. Plus, the Qt Simulator apparently cannot simulate the landscape mode of the N9, and it lies about the default font size of QML Text elements (if you do not specify a pixelSize, text elements will look all right in the simulator, but ridiculously tiny on the N9).

In the end I cobbled up a shell script that rsyncs my updated QML files to the device and runs a short Python script (over ssh) to launch them. You need rsync and PySide installed on the device for this, obviously, as well as having SSH set up for passwordless logins. As a bonus, I can now do QML development during lunch, directly on the device, with vim, enjoying proper syntax highlighting. :)

Oh, and my code is up on Github.

I started with the Ubuntu packages for FBReader, since they used a more sane build system for .debs (compared to upstream's funky shell script). Some tweaks were needed to make it build in Scratchbox: since GTK+ and Hildon libraries aren't available on Harmattan, I had to disable the building of -gtk and -maemo versions of libzlui. I also got to learn a new tool — quilt.

Fixing the segfault took a couple of days of debugging, studying the source code of both FBReader and Qt itself, and asking for help on IRC. Turns out FBReader was holding an active QPainter instance for too long, and its backing pixmap got destroyed (or, rather, converted from an OpenGL texture to a plain X11 pixmap) during a task switch, causing the crash. I'm probably describing this wrong BTW, but, in any case, adding QPainter::begin() and QPainter::end() calls in the paintEvent handler fixed the segfault.

Next, a small tweak in the .desktop file to make FBReader a single-instance application: change Exec=FBReader to Exec=single-instance /usr/bin/FBReader (I'm paraphrasing slightly).

Then, a more ambitious goal: making FBReader intercept volume keys and use them for scrolling. Google gave me a pointer to QmKeys, which was the wrong API to use here, but gave me a lead to qmkeyd2, which appears to be an open source daemon, which gave me a lead to sysuid, another open source daemon, which in turn gave me a lead to libresourceqt, and that was the right API at last.

Volume keys generate regular key events for XF86AudioRaiseVolume and XF86AudioLowerVolume, but they're also intercepted by qmkeyd2, which tells all subscribers (and sysuid is one) about them. Which subscriber gets to react is determined by the resource policy framework. So what I needed to do in FBReader was acquire the ScaleButtonResource when FBReader starts up or gets focus, and release it when FBReader quits or goes into background. That also required some IRC help until I discovered installEventFilter() and the ApplicationActivate/ApplicationDeactivate events. And QApplication::instance().

The various tools available in the developer firmware were invaluable: openssh, gdb, valgrind, strace, xev, xprop, lsof, netstat. Also, I would not have achieved my second goal without being able to look at the sources of Meego system components (qmkeyd, sysuid). Yay open source!

Here are my changes to the source code. You can find my modified Debian packaging files, as well as prebuilt binary packages (with full debug info, for gdb goodness), in my experimental harmattan apt repository. The UI is still ugly and non-native, but it doesn't matter much in fullscreen mode :) .

Note to self: when next building fbreader, make sure the 2 megabyte tags file doesn't end up in the .diff.gz. And speaking of crud in source packages, the vim package I built for Harmattan the other day contains the entire 50 meg .hg in the .orig.tar.gz. I need to figure out how to tell dh_make to omit it.

I intend to port GTimeLog to it. Although my more immediate need is to have FBReader, so that I can stop carrying both this one and my N900 with me everywhere. Also, vim would be nice.

I've already hacked up Lithuanian support to the virtual and hardware keyboards, thanks to the very nice design of Maliit. As a comparison, I've had my N900 for a year and a half, and I still can't type Lithuanian on it. XKB is not fun.

• shiny documentation built with Sphinx.
• Python 3.x support, thanks to Stefano Rivera.
• Python 2.4 and 2.5 support, thanks to Daniel Blackburn.
• assorted smaller improvements.

zodbbrowser got

• support for all ZODB databases, not just those with a Zope 3/Bluebream-style root folder/local site.
• the ability to cope better with broken objects (due to the way ZODB works, not having some of your modules on the Python pack can break unpickling; zodbbrowser now handles this kind of situation better).
• assorted smaller improvements.
• a slow but inevitable shift of focus from "use it as a plugin for your Zope 3 app" to "it's a standalone tool for inspecting ZODB contents". (Both use cases are still supported, and will be for the foreseeable future.)

imgdiff got

• its first public release.
• some experimental code to actually find and highlight the differing parts of the images:
  
  
  This works better when both images are the same size, although there's experimental (and somewhat buggy) code to try all possible alignments. I could use some help here; image processing is not something I'm familiar with, and searching StackOverflow didn't help beyond reminding me of the existence of PIL's ImageChops.difference(), which is for same-sized images only. Many of the results there are about comparing photos, where things like lighting levels matter. What I need is a diff for computer-generated images, where some things may be shifted around a bit, by different amounts, but are essentially the same. Are there any two-dimensional sequence diff algorithms?

Profiling

This WSGI middleware profiles every request with the cProfile module. To see the profiles, visit a hidden URL /_profiler/showall:

What you see here is heavily tweaked in my fork branch of Dozer; upstream version had no Cost column and didn't vary the background of Time by age (that last bit helps me see clumps of requests).

Here's what an individual profile looks like:

The call tree nodes can be expanded and collapsed by clicking on the function name. There's a hardcoded limit of 20 nesting levels (upstream had a limit of 15), sadly that appears not to be enough for practical purposes, especially if you start profiling Zope 3 applications...

You can also take a look at the WSGI environment:

Sadly, nothing about the response is captured by Dozer. I'd've liked to show the Content-Type and perhaps Content-Length in the profile list.

The incantation in development.ini is

[filter-app:profile]
use = egg:Dozer#profile
profile_path = /tmp/profiles
next = main

Create an empty directory /tmp/profiles and make sure other users cannot write to it. Dozer stores captured profiles as Python pickles, which are insecure and allow arbitrary command execution.

To enable the profiler, run paster like this:

$ paster serve development.ini -n profile

Bonus feature: call graphs

Dozer also writes a call graph in Graphviz "dot" format in the profile directory. Here's the graph corresponding to the profile you saw earlier, as displayed by the excellent XDot:

See the fork where the "hot" red path splits into two?

On the left we have Routes deciding to spend 120 ms (70% total time) recompiling its route maps. On the right we have the actual request dispatch. The actual controller action is called a bit further down:

Here it is, highlighted. 42 ms (24% total time), almost all of which is spent in SQLAlchemy, loading the model object (a 2515 byte image stored as a blob) from SQLite.

A mystery: pickle errors

When I first tried to play with the Dozer profiler, I was attacked by innumerable exceptions. Some of those were due to a lack of configuration (profile_path) or invalid configuration (directory not existing), or not knowing the right URL (going to /_profiler raised TypeError). I tried to make Dozer's profiler more forgiving or at least produce clearer error messages in my fork branch, e.g. going to /_profiler now displays the profile list.

However some errors were very mysterious: some pickles, written by Dozer itself, could not be unpickled. I added a try/except that put those at the end of the list, so you can see and delete them.

Does anybody have any clues as to why profile.py might be writing out broken pickles?

Update: as Ben says in the comments, my changes have been accepted upstream. Yay!

Log capturing

This WSGI middleware intercepts logging calls for every request. Here we see a toy Pylons application I've been working on in my spare time. Dozer added an info bar at the top:

When you click on it, you get to see all the log messages produced for this request. I've set SQLAlchemy's loglevel to INFO in my development.ini, which produces:

(Why on Earth does SQLAlchemy think I want to see the memory address of the Engine object in my log files, I don't know. The parentheses contain argument values for parametrized queries, of which there are none on this page.)

Upstream version displays absolute timestamps (of the YYYY-MM-DD HH:MM:SS.ssssss variety) in the first column; my fork shows deltas in milliseconds. The incantation in development.ini is

[filter-app:logview]
use = egg:Dozer#logview
next = main

which makes it disabled by default. To enable, you run paster like this:

$ paster serve development.ini -n logview

(Upstream version lacks the paste entry point for logview; it's in my fork, for which I submitted a pull request weeks ago like a good open-source citizen. Incidentally, patches for stuff I maintain have been known to languish for years in my inbox, so I'm not one to throw stones.)

Next: profiling with Dozer.

Update: Tom Longson blogged about this back in 2008! And his CSS is prettier.

In other news, logs2html now copies irclog.css to the destination directory (if it doesn't exist there already). I've been noticing logs produced with irclog2html on random places, and sometimes they were unstyled; hopefully this will become rare now.

Option 1: N900 as a USB modem

Use the provided USB cable to connect the N900 to the laptop. Choose "PC Suite" mode on the N900 when you get the USB connection menu. The laptop now sees your N900 as a bog-standard USB 3G modem. Use Network Manager to connect to the internet.

Pros: no extra setup required. The N900 and the laptop can both access the Internet at the same time.

Cons: you have to use a USB cable (I hate cables). You cannot ssh into your N900 (and ssh is my primary file transfer protocol between the laptop and the M900).

Option 2: N900 as a Bluetooth DUN modem

Install Bluetooth DUN support from Maemo Extras. Then use it like you would any other phone that has Bluetooth DUN.

Pros: no cables.

Cons: Bluetooth is the worst technology ever. I never had it work reliably. Plus, Network Manager in Ubuntu 10.04 doesn't support Bluetooth DUN (it supports only Bluetooth PAN, as far as I know).

Option 3: N900 as a WiFi access point with Joikuspot

I haven't tried this.

Pros: simple (hopefully), no cables required.

Cons: Joikuspot is non-free. I'm not an absolute zealot, but I will avoid closed-source stuff when open-source alternatives are available.

Option 4: N900 as a WiFi access point with Mobilehotspot

I haven't tried this either.

Pros: it's an open-source app available from Maemo Extras. No cables required.

Cons: requires a non-standard kernel (or so I've heard). Way outside my comfort level.

Option 5: N900 as a WiFi access point with shell scripts

Here's the shell script I run on my N900: share-wifi. It sets up an ad-hoc WiFi network, and starts a DHCP and DNS server (dnsmasq). Sadly, it cannot set up connection sharing (NAT), so I rely on OpenSSH as a SOCKS5 proxy. The whole setup is like this:

1. You want the latest firmware (PR 1.2) to avoid this bug.
2. You need to have OpenSSH installed on the N900. Also, setting up key-based authentication makes it more convenient.
3. The script assumes that you've set up sudo on the N900 so that you can run any command as root.
4. You need to have wireless-tools installed. It's in the main SSU repository so you should be able to sudo apt-get install it (if it's not preinstalled; I don't remember).
5. On the N900 run share-wifi in a terminal (optionally passing a WiFi channel number from 1 to 11, in case you need to avoid interference with nearby networks).
6. On the laptop connect to the new n900 WLAN and run ssh -D 1080 user@n900. You will get a shell session; the SOCKS proxy will be active while it is open.
7. Reconfigure your laptop to use a SOCKS5 proxy on localhost:1080. For GNOME systems I've a couple of shell scripts: proxy-on and proxy-off. For applications that do not use the GNOME proxy settings (such as Subversion access over SSH), use tsocks.
8. When done, hit Ctrl-C on the N900 to terminate the sharing script.

Pros: no non-free software or custom kernel required. No cables.

Cons: complicated to set up. No WLAN power savings available for ad-hoc networks, so battery life is extremely poor (~2 hours). But, hey, no cables!

...
  File "...", line ...
    from hashlib import md5
  File "/usr/lib/python2.6/hashlib.py", line 63, in __get_builtin_constructor
     import _md5
ImportError: No module named _md5

this means that the copy of the python executable in your virtualenv/bin directory is outdated and you should update it:

$ cp /usr/bin/python2.6 /path/to/venv/bin/python

or, better yet, recreate the virtualenv.

• re-partition or re-format the flash drive (this eliminates usb-creator, AFAIU)
• extract the contents of the ISO image into the root of the USB drive (this eliminates unetbootin)
• skip the ISO's bootloader and directly boot the kernel+initramfs from the ISO (eliminates this recipe, and this recipe)

I just want a bootloader on the USB to read the VFAT filesystem, mount the ISO image as a loop device, then chain-load the bootloader from that ISO. Bonus points for having a menu letting me choose one of several ISO images. Running a script to edit a text file (say, grub's config) to get that menu is fine.

Is this even possible? If not, can I at least have two out of three (no partition/extraction, but skipping intrinsic bootloader is fine)?

Solution that I finally chose (from Ubuntu forums):

Plug in USB key. Find out the mount point (/media/disk) and the device name (/dev/sdx) of the USB key with

$ mount|grep /media

Install GRUB 2 into the USB key with

$ sudo grub-install --root-directory=/media/usbdisk /dev/sdx

If it says something about embedding being impossible and falling back to UNRELIABLE blocklist-based setup, run

$ sudo grub-install --root-directory=/media/usbdisk /dev/sdx --force

Download a CD image, let's say ubuntu-10.10-desktop-i386.iso. Put it into /media/usbdisk/ubuntu/.

Create a text file /media/usbdisk/boot/grub/grub.cfg with

menuentry "Ubuntu 10.10 (x86 desktop livecd)" {
    set isofile="/ubuntu/ubuntu-10.10-desktop-i386.iso"
    loopback loop $isofile
    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile quiet splash noprompt --
    initrd (loop)/casper/initrd.lz
}

You can have as many ISO images as you want, just make sure to add a menuentry for each. There's no need to run grub-install again after adding or removing a .iso file. Oh, and if you want to use an ISO file for a different distribution, you'll have to figure out the correct linux and initrd lines somehow.

Update: Ubuntu 13.04 changed the name of the kernel -- use (loop)/casper/vmlinuz.efi instead of (loop)/casper/vmlinuz.

I tested it with the following images

• ubuntu-10.04-desktop-i386.iso
• ubuntu-10.04-server-i386.iso
• ubuntu-10.04-server-amd64.iso
• ubuntu-10.10-desktop-i386.iso
• ubuntu-11.04-desktop-i386.iso
• ubuntu-11.10-desktop-i386.iso
• ubuntu-12.04-desktop-i386.iso
• ubuntu-12.04-desktop-amd64.iso
• ubuntu-12.04-server-i386.iso
• ubuntu-12.04-server-amd64.iso
• ubuntu-12.10-desktop-i386.iso
• ubuntu-12.10-desktop-amd64.iso
• ubuntu-13.04-desktop-amd64.iso

This solution skips the CD-ROM's boot menu. I haven't found a better one.

Simple things should be easy; complicated things should be possible.

Frameworks make simple things easy. Good frameworks keep the complicated thing possible; poorly-designed frameworks make the complicated thing more difficult than necessary; bad frameworks make even simple things complicated.

Doing everything from scratch merely makes things possible, but rarely easy.

Grok is a Python web framework, built on top of the Zope Toolkit, which is the core of what used to be called Zope 3 and is now rebranded as BlueBream. Confused yet? Get used to it: the small pluggable components are the heart and soul of ZTK, and the source of its flexibility. It's not surprising that people take the same approach on a larger scale: take Zope 3 apart into smaller packages and reassemble them into different frameworks such as Grok, BlueBream or repoze.bfg.

The Grok book by Carlos de la Guardia introduces the framework by demonstrating how to create a small but realistic To-do list manager. I like this technique, and it works pretty well. The author covers many topics:

• creation of a new project
• simple views with Zope Page Templates
• automatic form generation from schemas (with tweaks)
• catalogs and indexes (my favourite chapter)
• security: users, roles, permissions; authentication and authorization
• extremely pluggable page layouts with viewlets and pagelets
• basic ZODB, blobs, ZEO, database packing, backups with repozo
• SQL databases, integration with SQLAlchemy (including a common transactional model)
• component architecture: adapters and utilities
• Martian: extending Grok by defining custom component directives
• very short intro to testing (zope.testing, unit tests and doctests, functional tests with zope.testbrowsing) and debugging (pdb; AJAXy debugger, which looks exactly like the Pylons one with an uglier skin)
• deployment (my second favourite chapter): paster, apache and mod_proxy, mod_wsgi, pound, squid, varnish, scalable deployments.

Some important topics like internationalization, time zones, testing with Selenium, and (especially) database migration (which is pretty specific for ZODB) were not covered.

If you want to learn about Grok, this book will be useful, but there's a caveat: there's the usual slew of typographical mistakes and other errors I've come to expect from books published by Packt. It's their third book I've seen; all three had surprisingly high numbers of errors. Some had more, others had fewer. The Grok book was on the high side and the first one where I was tempted to record a "WTFs per page" metric. The mistakes are easy to notice and correct, so they didn't impede my understanding of the book's content. Disclaimer: I've been working with Zope 3 for the last six-or-so years, so I was pretty familiar with the underlying technologies, just not the thin Grok convenience layer. If minor errors annoy you, stay away. I haven't noticed any major factual errors, although there were what I would consider some pretty important omissions:

• ZODB is not as transparent as people tell you. There are many gotchas, especially if you want to refactor your code without throwing away old databases.
• bin/buildout is free to recursively remove anything under parts. Keeping your database there is fine only if you don't mind occasionally starting from scratch.
• repozo does not back up blobs.
• The ZODB transaction conflict resolution depends on being able to repeat requests several times; this is important if your code has external side effects (e.g. sends emails, creates files, pings 3rd party websites over HTTP). Packages like megrok.rdb or zope.sendmail take care of this; it'd be nice to be shown how to do that for your own code before you discover this issue the hard way when your app starts charging people's credit cards three times every now and then.
• You need to make sure you send out object events at appropriate times, or your catalog indexes won't be updated.
• Permission and role grants are persistent: if you delete a user and then create a new one with the same username, the new user will have all the roles and permissions granted to the old one. If you implement user deletion, you need to explicitly remove old grants.
• The Zope security model expects every object to have a valid __parent__ attribute; permission/role grants will not work properly on objects without a __parent__. Most of the time this is taken care of automatically, but when it's not, you can get really confusing errors.
• applySkin should only be used for browser requests; blindly calling it from a traversal event handler can break WebDAV/XML-RPC. (Incidentally, I should file a bug about that; it should abort if you pass a non-browser request instead of silently converting it into a browser request.)
• Allowing end-users to specify ++skin++ in the URL can be a security hole.

Overall, Grok is pretty nice, especially compared to vanilla Zope 3. However, when compared to frameworks like Pylons or Django, Grok appears more complex and seemingly requires you to do additional work for unclear gain. For example, chapter 8 has you writing three components for every new form you add: one for the form itself, one for a pagelet wrapping the form, and one for a page containing the pagelet. Most of that code is very similar with only the names being different. I'm sure there are situations where this kind of extreme componentization pays off (e.g. it lets you override particular bits on particular pages to satisfy a particular client's requests, without affecting any other clients), but the book doesn't convincingly demonstrate those advantages. Again, I may be biased here since I've been enjoying those advantages for the past six years, without ever having felt the pain of doing similar customizations with a less flexible framework. (It's a gap in my professional experience that I'm itching to fill.)

Update: some other reviews on Planet Python.

Update 2: Another review (well, part 1 of one, but I got tired waiting for part 2).

Short summary: it's good book. I learned a thing or two from it. I don't know well it would work as an introductionary text for someone new to unit testing (or Python). Some of the bits seemed overcomplicated and underexplained, parts of the example code/tests seemed to contain design decisions received from mysterious sources.

Incidentally, Packt uses a simple yet effective method for watermarking e-books: my name and street address are displayed in the footer of every page. What's funny is that the two non-ASCII characters in the street name are replaced with question marks. It's not a data entry problem: the website that let me download those books shows my address correctly, so it must be happening somewhere in the PDF production process. I didn't expect this kind of Unicode buggyness from a publisher. Then again there were occasional strange little typographical errors in the text, like not leaving a space in front of an opening parenthesis in an English sentence, or using a never-seen-before +q= operator in Python code. I was also left wondering how the following sentence (page 225) could slip past the editing process:

doctest ignores everything between the Traceback (most recent last call).

Thankfully those small mistakes did not detract from the overall message of the book.

I liked the author's technique of showing subtly incorrect code, letting the reader look at it and miss all the bugs, and then showing how unit or integration tests catch the bugs the reader missed. I'm pretty sure there's at least one remaining bug that the author missed in the example package (storing a schedule doesn't erase old data), which could serve for a new chapter on regression testing if there's a second edition.

Summary of topics covered:

• Terms: unit testing, integration testing, system testing.
• Basics of doctest and unittest, their strengths and weaknesses.
• Using mocks (with Mocker).
• Using Nose.
• Test-Driven Development with lots of example code.
• Using Twill.
• Integration testing with lots of example code.
• Using coverage
• Post-commit hooks to run tests with Bazaar, Mercurial, Git, Darcs, Subversion.
• Continuous integration with Buildbot

I found the TDD cycle a bit larger than I generally like, but I believe it's a matter of taste, and perhaps a shorter cycle wouldn't work as well in a written medium.

I found it a bit jarring how the Twill chapter intrudes between the two chapters showing unit testing and integration testing of the same sample package. I think it would've been better to swap the order of chapters 8 and 9.

I liked the technique presented for picking subsets of the code for integration tests, although I wonder how well it would work on a larger project.

Topics not covered:

• Functional testing (which is very close but not exactly the same as system testing).
• Regression testing (page 46 contains advice about this without mentioning the term regression testing).
• Continuous integration with Hudson (simpler to set up than buildbot, easily covers 80% of cases).

As you can see these holes are all rather small.

Probably the biggest weakness of the book is the complexity of some things shown:

• writing mocks for pure unit tests
• mocking other instances of the same class under test
• even occasionally mocking self, which needs tricks like calling a method's im_func directly
• mocking __reduce_ex__ so you can pickle mocks in an integration test, instead of using real classes or simple stubs.
• testing the same code multiple times: unit tests, several sets of integration tests that test ever-increasing subsets of classes
• Buildbot instead of Hudson

Seeing the repetitive and redundant mock code in the first few doctest examples I started asking what's the point?, but the book failed to provide a compelling answer (the answer provided—it's easier to locate bugs—works just as well for integration tests that focus on individual classes). And there are good answers for that question, like instant feedback from your unit test suite. Are they worth the additional development effort? Maybe that depends on the developer. I don't think they would help me, so I tend to stick with low-level integration tests I call "unit tests" (as well as system tests; it's always a mistake to keep all your tests in a single level). I'm slightly worried that this book might give the wrong impression (testing is hard) and turn away beginning Python programmers from writing tests altogether.

Overall I do not feel that I have wasted my time reading Python Testing. I look forward to reading the other reviews that showed up on Planet Python. I gathered that not all reviewers were happy with the book, but avoided reading their reviews in order not to influence my own.

Update: I especially liked this review by Brian Jones. The lack of awkward page breaks in code examples is something that I only noticed after reading a different book, which is full of such awkward breaks, sigh.

Update 2: The book links are now affiliate links; I get a small amount from any purchase you make through them.

Profiling a Python program is getting easier and easier:

$ python -m cProfile -o prof.data bin/test -f

runs our test runner (which is a Python script) under the profiler and stores the results in prof.data.

$ runsnake prof.data

launches the RunSnakeRun profile viewer, which displays the results visually:

The square map display of RunSnakeRun, with the 'render_restructured_text' function highlighted.

Who knew that ReStructuredText rendering could be such a time waster? A short caching decorator and the test suite is twice as fast. The whole exercise took me less than an hour. I should've done it sooner.

Other neat tools:

• pstats from the standard library lets you load and display profiler results from the command line (try python -m pstats prof.data).
• pyprof2calltree converts Python profiler data files to a format that the popular profiler visualization tool kcachegrind can understand. It's somewhat less useful now that RunSnakeRun exists.
• profilehooks by yours truly has decorators for easily profiling individual functions instead of entire scripts.
• keas.profile and repoze.profile hook up the profiler as WSGI middleware for easy profiling of web apps.