I've been testing (as well as writing) Python code for the last eight years, so a book with the words Begginer's Guide prominently displayed on the cover isn't something I'd've decided to buy for myself. Nevertheless I jumped at the offer of receiving a free e-copy for reviewing it.
Short summary: it's good book. I learned a thing or two from it. I don't know well it would work as an introductionary text for someone new to unit testing (or Python). Some of the bits seemed overcomplicated and underexplained, parts of the example code/tests seemed to contain design decisions received from mysterious sources.
Incidentally, Packt uses a simple yet effective method for watermarking e-books: my name and street address are displayed in the footer of every page. What's funny is that the two non-ASCII characters in the street name are replaced with question marks. It's not a data entry problem: the website that let me download those books shows my address correctly, so it must be happening somewhere in the PDF production process. I didn't expect this kind of Unicode buggyness from a publisher. Then again there were occasional strange little typographical errors in the text, like not leaving a space in front of an opening parenthesis in an English sentence, or using a never-seen-before +q= operator in Python code. I was also left wondering how the following sentence (page 225) could slip past the editing process:
doctest ignores everything between the Traceback (most recent last call).
Thankfully those small mistakes did not detract from the overall message of the book.
I liked the author's technique of showing subtly incorrect code, letting the reader look at it and miss all the bugs, and then showing how unit or integration tests catch the bugs the reader missed. I'm pretty sure there's at least one remaining bug that the author missed in the example package (storing a schedule doesn't erase old data), which could serve for a new chapter on regression testing if there's a second edition.
Summary of topics covered:
- Terms: unit testing, integration testing, system testing.
- Basics of doctest and unittest, their strengths and weaknesses.
- Using mocks (with Mocker).
- Using Nose.
- Test-Driven Development with lots of example code.
- Using Twill.
- Integration testing with lots of example code.
- Using coverage
- Post-commit hooks to run tests with Bazaar, Mercurial, Git, Darcs, Subversion.
- Continuous integration with Buildbot
I found the TDD cycle a bit larger than I generally like, but I believe it's a matter of taste, and perhaps a shorter cycle wouldn't work as well in a written medium.
I found it a bit jarring how the Twill chapter intrudes between the two chapters showing unit testing and integration testing of the same sample package. I think it would've been better to swap the order of chapters 8 and 9.
I liked the technique presented for picking subsets of the code for integration tests, although I wonder how well it would work on a larger project.
Topics not covered:
- Functional testing (which is very close but not exactly the same as system testing).
- Regression testing (page 46 contains advice about this without mentioning the term regression testing).
- Continuous integration with Hudson (simpler to set up than buildbot, easily covers 80% of cases).
As you can see these holes are all rather small.
Probably the biggest weakness of the book is the complexity of some things shown:
- writing mocks for pure unit tests
- mocking other instances of the same class under test
- even occasionally mocking self, which needs tricks like calling a method's im_func directly
- mocking __reduce_ex__ so you can pickle mocks in an integration test, instead of using real classes or simple stubs.
- testing the same code multiple times: unit tests, several sets of integration tests that test ever-increasing subsets of classes
- Buildbot instead of Hudson
Seeing the repetitive and redundant mock code in the first few doctest examples I started asking what's the point?, but the book failed to provide a compelling answer (the answer provided—it's easier to locate bugs—works just as well for integration tests that focus on individual classes). And there are good answers for that question, like instant feedback from your unit test suite. Are they worth the additional development effort? Maybe that depends on the developer. I don't think they would help me, so I tend to stick with low-level integration tests I call "unit tests" (as well as system tests; it's always a mistake to keep all your tests in a single level). I'm slightly worried that this book might give the wrong impression (testing is hard) and turn away beginning Python programmers from writing tests altogether.
Overall I do not feel that I have wasted my time reading Python Testing. I look forward to reading the other reviews that showed up on Planet Python. I gathered that not all reviewers were happy with the book, but avoided reading their reviews in order not to influence my own.
Update: I especially liked this review by Brian Jones. The lack of awkward page breaks in code examples is something that I only noticed after reading a different book, which is full of such awkward breaks, sigh.
Update 2: The book links are now affiliate links; I get a small amount from any purchase you make through them.
