Random notes from mg

Some anonymous Planet Python poster (at least I couldn't find the author's name on the blog) Christian Wyglendowski asks about a surprising difference between old-style and new-style classes. Since the comments on their blog are closed (which you find out only after pressing Submit), I'll answer here.

The question, slightly paraphrased: given a class

class LameContainerOld:
    def __init__(self):
        self._items = {'bar':'test'}
 
    def __getitem__(self, name):
        return self._items[name]
 
    def __getattr__(self, attr):
        return getattr(self._items, attr)

why does the 'in' operator work

>>> container = LameContainerOld()
>>> 'foo' in container
False
>>> 'bar' in container
True

when the equivalent new-style class raises a KeyError: 0 exception? Also, why does __getattr__ appear to be called to get the bound __getitem__ method of the dict?

>>> container.__getitem__
<bound method LameContainerNew.__getitem__ of {'bar': 'test'}>

What actually happens here is that LameOldContainer.__getattr__ gets called for special methods such as __contains__ and __repr__. This is why (1) the 'in' check works, and (2) it appears, at first glance, that you get the wrong __getitem__ bound method. If you pay close attention to the output, you'll see that it's the __getitem__ of LameOldContainer; it's just that repr(LameOldContainer()) gets proxied through to the dict.__repr__ when you don't expect it:

>>> container
{'bar': 'test'}

Special methods never go through __getattr__ for new-style classes, therefore neither __contains__ nor __repr__ are proxied if you make the container inherit object. If there's no __contains__ method, Python falls back to the sequence protocol and starts calling __getitem__ for numbers 0 through infinity, or until it gets an IndexError exception.

I've just spent the whole night setting up blog comments. PyBlosxom doesn't make it painless, sadly, more like the opposite.

First: don't be scared by the list of comment-related plugins on the PyBlosxom site. There's only one important plugin: comments. All others depend on it and enhance its functionality. The last three or four times I was about to add comments to my blog I got scared at step one: evaluate the available plugins. Don't repeat my mistake!

Second, follow the instructions carefully. There's no shortcut.

Third, fix what's broken. Be prepared to debug the source code. print >> sys.stderr, "message" is your friend.

Fourth, fiddle with the look (CSS and HTML).

Fifth, write a blog post and eagerly await your first comments.

Step 3 screams for an explanation, doesn't it? Problem 1: the comments plugin requires that you use categories in your blog. I'm not (I'm holding out for tags). Workaround: comment out if entry['absolute_path'] check in cb_story and cb_story_end.

Problem 2: the AJAX post returns "Empty response from server". Workaround: modify cb_story_end to call readComments directly if entry['num_comments'] is None, since cb_story, which usually does the read, is not called during the AJAX post.

Problem 3: if you enable comment moderation (by setting comment_draft_ext to a different value from comment_ext), the AJAX post returns "Empty response from server" once more. Workaround: modify cb_prepare to notice this case and set data['moderated'] = True, create a new template comment-moderated and render it in cb_story_end just like the preview template is rendered; also modify __shouldOutput to return True when rendering comment-moderated.

I'll post patches to the pyblosxom mailing tomorrow, unless I forget. It's 6 am already, and I'm kind of sleepy. I just hope I haven't inadvertently broken my RSS feed or flooded any planets.

Oh, and a helpful hint: don't name the post you're writing comments.txt, or the #comments anchor will point to the start of the story instead of the comments.

Update: The story continues, but solution is not in sight yet.

I upgraded a buildbot slave to Ubuntu 8.04 (Hardy) recently and now I'm getting a strange intermittent failure: sometimes cp -r /local/dir /nfs/mounted/dir fails ("process killed by signal 1", i.e. SIGHUP).

I wonder if NFS is relevant or incidental to the issue?

Google finds an old thread from 2005, with a workaround (usepty=False), but I'd like to understand the problem before applying random fixes.

So far three different build steps doing cp -r have failed during 10 days. I've now changed them all to cp -rv, so I can at least see if the failure is in the middle of the copy or at the end, if it fails again.

Update: so far 4 build steps have failed on 6 separate occasions:

May  5 02:31: cp -r local-dir1 nfs-mounted-dir1  
May  6 02:31: cp -r local-dir1 nfs-mounted-dir1  
May  6 04:33: cp -r local-dir2 nfs-mounted-dir2  
May 15 02:00: cp -r local-dir3 nfs-mounted-dir3  
May 17 04:32: rm -rf nfs-mounted-dir4            
May 20 04:31: rm -rf nfs-mounted-dir4            

I see no particular correlation between step duration and results, e.g. the rm -rf step usually takes between 2.2 and 4.6 seconds. The two SIGHUPs happened after 2.4 seconds.

They all make no output. When I changed the cp steps and added a -v, they stopped failing, but that could be just a coincidence.

We're having an email conversation with Jean-Paul Calderone ("exarkun") about the possibility of this being PTY-related, with no clear resolution so far.

And, hey, now this blog supports comments ;)

It's been a while since the last Expert Python Programming review on Planet Python. Y'all might've forgotten about this book by now. Time for a reminder? (Actually, I'm just lazy busy, and this is why this review hasn't appeared sooner.)

I received a free PDF copy of this book from Packt Publishing, with the understanding that I'll post a review on my blog. This is it. Short summary: it is a good book marred by a lot of mostly inconsequential little mistakes. I'd give it four stars out of five.

Aside: the PDF that I could download was personalized and had my name and address in the footer of every page. A very nice form of DRM that did not restrict my software choices for reading the book (Evince and also PDF Reader on Nokia Internet Tablets).

I bring it up here because it seems that Packt could've also applied fixes for the known errata to the personalized version, yet missed that opportunity. Perhaps it's technically more difficult than slapping a footer on every page. Or maybe it's better if everyone buying the book, whether in paper or in PDF, gets to see the same text.

The author (Tarek Ziade) covers a wide range of topics in the book, ranging from syntax (probably useful for those who've been programming in Python for quite a few years, and didn't have the time to keep up with the language changes before picking up this book) to style, source code organization, project infrastructure, software life cycle, documentation, testing and optimization, and finally ending with a review of some of the popular design patterns. The middle parts were the most interesting for me personally. I learned a thing or two, disagreed with the author on a few minor points (which are mostly a matter of preference), and managed to finish the book despite constant irritating little pricks I feel when I notice an error (I confess I'm a pedant. A missing space after a colon drives me up the wall).

As an example of the disagreement: I have an aversion to code-generating tools where you have to edit the generated code by hand. I could say more, but this is a topic for another time. Next, I strongly dislike sudo easy_install since it scribbles onto the part of the filesystem exclusively reserved for your OS's package management tools. And I don't think porting the original 23 design patterns to other programming languages is a good way to describe what those languages are about. (Also, set tabstop=4 in your .vimrc? Heresy! The Right Thing To Do is set softtabstop=4, as all right-thinking Vim users will doubtlessly agree. All hail the one true text editor! Oh dear, now I'm glad I don't have comments on this blog...)

The goodies: Chapter 1 (the bits about PYTHONSTARTUP on page 19) gave me persistent history for my interactive Python prompt, nicely complementing the coloured prompt and tab-completion I already had snarfed from somewhere else on the net (probably Peter Norvig's Python IAQ). Chapter 12 provided good examples of how to do profiling for time (page 281) and memory (page 291). I like Tarek's @profile decorator (measure time, pystones and memory at the same time). My profilehooks module was not mentioned, *sniff* ;-). Chapter 13 told me about Queue.join and task_done that snuck into the stdlib with Python 2.5 without me noticing.

I haven't mentioned topics covered in the book that I was already familiar with, such as setuptools, virtualenv, zc.buildout, Sphinx, Nose, Buildbot, or Mercurial. Yet, in my opinion, those are the most useful parts of the book. The breadth of the topics is amazing: I could hardly think of something that every serious Python programmer should know that isn't wasn't mentioned. I believe the depth was exactly right: mention solutions that are available, show how they feel when used and what they can do, point to the relevant web page and then stop. And not only tools, the descriptions of workflows (how to organize your source trees, how to develop software consisting of multiple packages, how to make releases), while hardly universal, are invaluable.

One thing prevents this from being a perfect book: errata. At around page 95, according to my notes, I invented a new metric of book quality: WTFs per page, It's closely related to WTFs per minute, but independent of your reading speed. At around page 165 I got tired of making a note of every little thing that I noticed and started just reading. This was considerably more enjoyable. I hope there's a second edition will all the bugs shaken out. To that end, I should go through my notes again and submit them via the online errata form. Yay, more work...