Random notes from mg
Sun, 04 Apr 2010
Disclaimer: I received a free review copy of this book. The book links are
affiliate links; I get a small amount from any purchase you make through them.
Grok is a Python web framework, built on
top of the Zope Toolkit, which is the core of what used to be called Zope 3 and
is now rebranded as BlueBream. Confused yet? Get used to it: the small
pluggable components are the heart and soul of ZTK, and the source of its
flexibility. It's not surprising that people take the same approach on a larger
scale: take Zope 3 apart into smaller packages and reassemble them into
different frameworks such as Grok, BlueBream or repoze.bfg.
book by Carlos de la Guardia introduces the framework by demonstrating
how to create a small but realistic To-do list manager. I like this technique,
and it works pretty well. The author covers many topics:
- creation of a new project
- simple views with Zope Page Templates
- automatic form generation from schemas (with tweaks)
- catalogs and indexes (my favourite chapter)
- security: users, roles, permissions; authentication and authorization
- extremely pluggable page layouts with viewlets and pagelets
- basic ZODB, blobs, ZEO, database packing, backups with repozo
- SQL databases, integration with SQLAlchemy (including a common
- component architecture: adapters and utilities
- Martian: extending Grok by defining custom component directives
- very short intro to testing (zope.testing, unit tests and doctests,
functional tests with zope.testbrowsing) and debugging (pdb; AJAXy
debugger, which looks exactly like the Pylons one with an uglier skin)
- deployment (my second favourite chapter): paster, apache and mod_proxy,
mod_wsgi, pound, squid, varnish, scalable deployments.
Some important topics like internationalization, time zones, testing with
Selenium, and (especially) database migration (which is pretty specific for
ZODB) were not covered.
If you want to learn about Grok, this book will be useful,
but there's a caveat: there's the usual slew of typographical mistakes and
other errors I've come to expect from books published by Packt. It's their
third book I've seen; all three had surprisingly high numbers of errors. Some
had more, others had fewer. The Grok book was on the high side and the first
one where I was tempted to record a "WTFs per page" metric.
The mistakes are easy to notice and correct, so they didn't impede my
understanding of the book's content. Disclaimer: I've been working with
Zope 3 for the last six-or-so years, so I was pretty familiar with the
underlying technologies, just not the thin Grok convenience layer. If
minor errors annoy you, stay away. I haven't noticed any major
factual errors, although there were what I would consider some pretty important
- ZODB is not as transparent as people tell you. There are many gotchas,
especially if you want to refactor your code without throwing away old
- bin/buildout is free to recursively remove anything under
parts. Keeping your database there is fine only if you don't mind
occasionally starting from scratch.
- repozo does not back up blobs.
- The ZODB transaction conflict resolution depends on being able to
repeat requests several times; this is important if your code has external
side effects (e.g. sends emails, creates files, pings 3rd party websites over
HTTP). Packages like megrok.rdb or zope.sendmail take care of this; it'd be
nice to be shown how to do that for your own code before you discover this
issue the hard way when your app starts charging people's credit cards three
times every now and then.
- You need to make sure you send out object events at appropriate times, or
your catalog indexes won't be updated.
- Permission and role grants are persistent: if you delete a user and then
create a new one with the same username, the new user will have all the roles
and permissions granted to the old one. If you implement user deletion, you
need to explicitly remove old grants.
- The Zope security model expects every object to have a valid
attribute; permission/role grants will not work properly on objects without a
__parent__. Most of the time this is taken care of
automatically, but when it's not, you can get really confusing errors.
applySkin should only be used for browser requests; blindly
calling it from a traversal event handler can break WebDAV/XML-RPC.
(Incidentally, I should file a bug about that; it should abort if you pass a
non-browser request instead of silently converting it into a browser
- Allowing end-users to specify
++skin++ in the URL can be a
Overall, Grok is pretty nice, especially compared to vanilla Zope 3.
However, when compared to frameworks like Pylons or Django, Grok appears more
complex and seemingly requires you to do additional work for unclear gain. For
example, chapter 8 has you writing three components for every new form you add:
one for the form itself, one for a pagelet wrapping the form, and one for a
page containing the pagelet. Most of that code is very similar with only the
names being different. I'm sure there are situations where this kind of
extreme componentization pays off (e.g. it lets you override particular bits on
particular pages to satisfy a particular client's requests, without affecting
any other clients), but the book doesn't convincingly demonstrate those
advantages. Again, I may be biased here since I've been enjoying those
advantages for the past six years, without ever having felt the pain of doing
similar customizations with a less flexible framework. (It's a gap in my
professional experience that I'm itching to fill.)
Update: some other
on Planet Python.
Update 2: Another
review (well, part 1 of one, but I got tired waiting for part 2).
Sat, 13 Mar 2010
I've been testing (as well as writing) Python code for the last eight years,
so a book with the words Begginer's Guide prominently displayed on
the cover isn't something I'd've decided to buy for myself. Nevertheless
I jumped at the offer of receiving a free e-copy for reviewing it.
Short summary: it's good book. I learned a thing or two
from it. I don't know well it would work as an introductionary text for
someone new to unit testing (or Python). Some of the bits seemed
overcomplicated and underexplained, parts of the example code/tests seemed to
contain design decisions received from mysterious sources.
Incidentally, Packt uses a simple yet
effective method for watermarking e-books: my name and street address are
displayed in the footer of every page. What's funny is that the two non-ASCII
characters in the street name are replaced with question marks. It's not a
data entry problem: the website that let me download those books shows my
address correctly, so it must be happening somewhere in the PDF production
process. I didn't expect this kind of Unicode buggyness from a publisher.
Then again there were occasional strange little typographical errors in the
text, like not leaving a space in front of an opening parenthesis in an English
sentence, or using a never-seen-before +q= operator in Python code. I
was also left wondering how the following sentence (page 225) could slip past
the editing process:
doctest ignores everything between the Traceback (most recent last call).
Thankfully those small mistakes did not detract from the overall message of
I liked the author's technique of showing subtly incorrect code, letting the
reader look at it and miss all the bugs, and then showing how unit or
integration tests catch the bugs the reader missed. I'm pretty sure there's at
least one remaining bug that the author missed in the example package (storing
a schedule doesn't erase old data), which could serve for a new chapter on
regression testing if there's a second edition.
Summary of topics covered:
- Terms: unit testing, integration testing, system testing.
- Basics of doctest and unittest, their strengths and weaknesses.
- Using mocks (with Mocker).
- Using Nose.
- Test-Driven Development with lots of example code.
- Using Twill.
- Integration testing with lots of example code.
- Using coverage
- Post-commit hooks to run tests with Bazaar, Mercurial, Git, Darcs,
- Continuous integration with Buildbot
I found the TDD cycle a bit larger than I generally like, but I believe it's
a matter of taste, and perhaps a shorter cycle wouldn't work as well in a
I found it a bit jarring how the Twill chapter intrudes between the two
chapters showing unit testing and integration testing of the same sample
package. I think it would've been better to swap the order of chapters 8 and
I liked the technique presented for picking subsets of the code for
integration tests, although I wonder how well it would work on a larger
Topics not covered:
- Functional testing (which is very close but not exactly the same as
- Regression testing (page 46 contains advice about this without mentioning
the term regression testing).
- Continuous integration with Hudson (simpler to set up than buildbot,
easily covers 80% of cases).
As you can see these holes are all rather small.
Probably the biggest weakness of the book is the complexity of some
- writing mocks for pure unit tests
- mocking other instances of the same class under test
- even occasionally mocking self, which needs tricks like
calling a method's im_func directly
- mocking __reduce_ex__ so you can pickle mocks in an
integration test, instead of using real classes or simple
- testing the same code multiple times: unit tests, several sets of
integration tests that test ever-increasing subsets of classes
- Buildbot instead of
Seeing the repetitive and redundant mock code in the first few doctest
examples I started asking what's the point?, but the book failed to
provide a compelling answer (the answer provided—it's easier to locate
bugs—works just as well for integration tests that focus on individual
classes). And there are good answers for that question, like instant feedback
from your unit test suite. Are they worth the additional development effort?
Maybe that depends on the developer. I don't think they would help me, so I
tend to stick with low-level integration tests I call "unit tests" (as well as
system tests; it's always a mistake to keep all your tests in a single level).
I'm slightly worried that this book might give the wrong impression (testing is
hard) and turn away beginning Python programmers from writing tests
Overall I do not feel that I have wasted my time reading Python
Testing. I look forward to reading the
that showed up on Planet Python. I gathered that not all reviewers were happy
with the book, but avoided reading their reviews in order not to influence my
Update: I especially liked this
review by Brian Jones. The lack of awkward page breaks in code examples
is something that I only noticed after reading a different book, which is full
of such awkward breaks, sigh.
Update 2: The book links are now affiliate links; I get a small
amount from any purchase you make through them.
Fri, 08 May 2009
It's been a while since the last Expert Python
Programming review on Planet
Python. Y'all might've forgotten about this book by now. Time for a
reminder? (Actually, I'm just lazy busy, and this is why this review hasn't appeared
I received a free PDF copy of this book from Packt Publishing, with the
understanding that I'll post a review on my blog. This is it. Short summary:
it is a good book marred by a lot of mostly inconsequential little mistakes.
I'd give it four stars out of five.
Aside: the PDF that I could download was personalized and had my
name and address in the footer of every page. A very nice form of DRM that
did not restrict my software choices for reading the book (Evince and also
PDF Reader on Nokia Internet Tablets).
I bring it up here because it seems that Packt could've also applied fixes
known errata to the personalized version, yet missed that opportunity.
Perhaps it's technically more difficult than slapping a footer on every page.
Or maybe it's better if everyone buying the book, whether in paper or in
PDF, gets to see the same text.
The author (Tarek Ziade) covers a wide range of topics in the book, ranging
from syntax (probably useful for those who've been programming in Python for
quite a few years, and didn't have the time to keep up with the language
changes before picking up this book) to style, source code organization,
project infrastructure, software life cycle, documentation, testing and
optimization, and finally ending with a review of some of the popular design
patterns. The middle parts were the most interesting for me personally. I
learned a thing or two, disagreed with the author on a few minor points (which
are mostly a matter of preference), and managed to finish the book despite
constant irritating little pricks I feel when I notice an error (I confess I'm
a pedant. A missing space after a colon drives me up the wall).
As an example of the disagreement: I have an aversion to code-generating
tools where you have to edit the generated code by hand. I could say more, but
this is a topic for another time. Next, I strongly dislike sudo
easy_install since it scribbles onto the part of the filesystem
exclusively reserved for your OS's package management tools. And I don't think
porting the original 23 design patterns to other programming languages is a
good way to describe what those languages are about. (Also, set
tabstop=4 in your .vimrc? Heresy! The Right Thing To Do is
set softtabstop=4, as all right-thinking Vim users will
doubtlessly agree. All hail the one true text editor! Oh dear, now I'm
glad I don't have comments on this blog...)
The goodies: Chapter 1 (the bits about PYTHONSTARTUP
on page 19) gave me persistent history for my interactive Python prompt, nicely
complementing the coloured prompt and tab-completion I already had snarfed from
somewhere else on the net (probably Peter Norvig's Python IAQ). Chapter 12
provided good examples of how to do profiling for time (page 281) and memory
(page 291). I like Tarek's @profile decorator (measure time, pystones
and memory at the same time). My profilehooks module was
not mentioned, *sniff* ;-). Chapter 13 told me about Queue.join
that snuck into the stdlib with Python 2.5 without me noticing.
I haven't mentioned topics covered in the book that I was already familiar
with, such as setuptools, virtualenv, zc.buildout, Sphinx, Nose, Buildbot, or
Mercurial. Yet, in my opinion, those are the most useful parts of the book.
The breadth of the topics is amazing: I could hardly think of something that
every serious Python programmer should know that isn't wasn't mentioned. I
believe the depth was exactly right: mention solutions that are available, show
how they feel when used and what they can do, point to the relevant web page
and then stop. And not only tools, the descriptions of workflows (how to
organize your source trees, how to develop software consisting of multiple
packages, how to make releases), while hardly universal, are invaluable.
One thing prevents this from being a perfect book: errata. At around page
95, according to my notes, I invented a new metric of book quality: WTFs per
page, It's closely related to WTFs per minute, but
independent of your reading speed. At around page 165 I got tired of making a
note of every little thing that I noticed and started just reading. This was
considerably more enjoyable. I hope there's a second edition will all the bugs
shaken out. To that end, I should go through my notes again and submit them
via the online errata form. Yay, more work...