Yesterday I slashed 50% of run time from our applications functional test
suite by modifying a single function. I had no idea that function was
responsible for 50% of the run time until I started profiling.
Profiling a Python program is getting easier and easier:
$ python -m cProfile -o prof.data bin/test -f
runs our test runner (which is a Python script) under the profiler and stores
the results in prof.data.
$ runsnake prof.data
launches the RunSnakeRun
profile viewer, which displays the results visually:
The square map display of RunSnakeRun, with the 'render_restructured_text'
function highlighted.
Who knew that ReStructuredText rendering could be such a time waster? A
short caching decorator and the test suite is twice as fast. The whole
exercise took me less than an hour. I should've done it sooner.
Other neat tools:
- pstats
from the standard library lets you load and display profiler results from the
command line (try python -m pstats prof.data).
- pyprof2calltree
converts Python profiler data files to a format that the popular profiler
visualization tool kcachegrind
can understand. It's somewhat less useful now that RunSnakeRun exists.
- profilehooks by
yours truly has decorators for easily profiling individual functions instead
of entire scripts.
- keas.profile and repoze.profile hook
up the profiler as WSGI middleware for easy profiling of web apps.
Things I've taken up to do in the nearest future:
-
Read and review Python
Testing: Beginner's Guide and Grok
1.0 Web Development for Packt. (The links are trackable to my blog,
but I'm not getting anything out of it. Other than free copies of the
e-books, which I already received, in exchange for a promise to review them
on this blog.)
-
Help Reportlab
folks set up continuous integration (most likely Hudson, since Buildbot, while powerful, has a steep
learning curve).
-
Think about becoming the buildbotmaster for Zope. Originally I intended
to volunteer to set up a few buildbots for various Zopeish projects
(ZTK, BlueBream, Grok, Zope 2) since half of the existing
ones were down or broken. Then various people fixed some of the
broken ones and other people chimed in mentioning existing buildbots that
nobody else knew about. There is a need for somebody to coordinate all
this activity: make sure we have up-to-date test results for all kinds of
projects, aggregate them in one place, chase up build slaves for exotic
OSes (i.e. Windows)... I don't think I'm well suited for this kind of
organisational activity.
-
Push along the various scratch-my-itch open source projects (GTimeLog, irclog2html,
zodbbrowser).
-
No idea what, but I've been wanting to do something for Maemo. Something small, given the copious
amounts of free time I have.
-
Then there's the paying work. On the plus side, there are opportunities
for fun there (today I slashed functional test run time by a half, by
adding a small caching decorator in front of a single function.
RunSnakeRun
and cProfile
rule!)
-
You know what, scratch the Zope buildbotmaster idea. Maybe I can do
something technical there, e.g. a cron script to ping the various buildbot,
scrape HTML/parse emails and aggregate build results. Maybe.
-
I hope I don't get burnout
again. Because that would suck. Again. Been there, done that, didn't
even get a T-shirt.
I really ought to read Getting Things Done. Reading it has been on my
todo-list for years.
Sorry for flooding Planet Maemo -- it was a side effect of changing this
feed's URL to only include posts tagged "maemo". I'm not sure if the fault
is PyBlosxom's or the aggregator's
As a penance, here's a Terminal trick for you:
LABELS='[Tab,Esc,Enter,PgUp,PgDn,F2,VKB]'
KEYS='[Tab,Escape,KP_Enter,Page_Up,Page_Down,F2,Return]'
gconftool -s /apps/osso/xterm/key_labels --type list --list-type string "$LABELS"
gconftool -s /apps/osso/xterm/keys --type list --list-type string "$KEYS"
This changes the toolbar to have three extra keys (Enter, F2, and a key that
acts like Enter when the hardware keyboard is open, and opens the virtual
keyboard if the hardware keyboard is closed).
Update: added screenshot:
Nokia N900 Terminal app with new toolbar buttons
On Tuesday we started what will hopefully become a tradition: weekly IRC
meetings for Zope developers. Topics covered include buildbot organization and
maintenance, open issues with the ZTK development process, and the fate of Zope
3.5 (= BlueBream 1.0).
There are IRC logs of the meeting, and Christian Theune posted a summary
to the mailing list.
My take on this can be summed up as: Zope ain't dead yet! The project has
fragmented a bit (Zope 2, Zope Toolkit, Grok, BlueBream, Repoze), but we all
share a set of core packages and we want to keep them healthy.
Next meeting is also happening on a Tuesday, at 15:00 UTC on #zope in
FreeNode.
Michael Foord wrote about some
Latin-1 control character fun in a blog that's hard to read (the RSS feed
syndicated on Planet Python is truncated, grr!) and hard to reply (no comments
on the blog! my Chromium's AdBlock+ hid the comment link so I couldn't
find it), but never mind that.
Unfortunately the data from the customers included some \x85 characters,
which were breaking the CSV parsing.
0x85 is a control character (NEXT LINE or NEL) in Latin-1, but it's a
printable character (HORIZONTAL ELLIPSIS) in Microsoft's code page 1252, which
is often mistaken for Latin-1. I would venture a suggestion that the encoding
of the customer data was not latin-1 but rather cp1252.
>>> '\x85'.decode('cp1252')
u'\u2026'
