The second Maemo Summit is over.

Nokia surprised everyone on the first day by handing out 300 pre-release N900s to the participants. I'm so happy now that after a long period of wavering I finally decided to come to the summit! The device is much better than I expected/feared (and I haven't even put a SIM card in yet). We're supposed to provide feedback and will have to send the devices back to Nokia in 6 months. (Nokia insisted on loan contracts signed in blood, kidding, but there are contracts.)

The tiny pixels are beautiful. It's what, 266 pixels per inch? Even older 225 dpi devices spoiled me: both the first generation iPhone and the first generation Kindle displays seemed very coarse and pixellated.

The user interface is very smooth. Having a composition manager improves apparent responsiveness: even if the app is swapped out and not ready to redraw, switching between windows appears to be instant since the picture is cached. And there's no flicker while the apps are redrawing. (Flickering during redraw is one of the main reasons I did not buy a S60 phone and stayed with good old S40.) Speaking of swapping, it's barely noticeable. You can run more apps than fit in RAM without having to suffer. The flash memory is noticeably faster than in a N810. And there's more of it (32 gigs: 28 gig partition for user data, the rest for the system: swap, applications, config files, etc.)

The design and usability of the user interface have improved a lot since the N810. The UI is pretty. Many of the apps are now convenient to use. Pervasive kinetic scrolling is sweet (except when you have really long lists or web pages, then it takes forever to reach the end).

Finally there are PIM-y things people missed in older Maemo releases: calendaring, contacts that can record all kinds of information (such as phone numbers).

All right, enough gushing. There were some irritating things too. For example, Bluetooth support is buggy/incomplete in the pre-release firmware, so it's hard to transfer files. Calendar/contacts sync with S40 phones does not work either. GPS is utterly useless when you're offline (no maps, or at least I haven't found a way to pre-download and cache them; also very long fix times without network assistance). Since I have no desire to pay extortionist roaming charges of my provider (2.5 EUR per megabyte), and haven't had a chance to go look for a prepaid SIM card, I usually have either WiFi or GPS coverage, but not both.

As you can guess, playing the device diverted a part of my attention from the presentations somewhat. I tried to compensate for that by reporting on the talks on IRC (using xchat on the device). I think the strategy backfired; IRC is rather disruptive and the channel is quite busy lately.

I booked a stay at a particular hotel because the web page said "Free WiFi". It didn't say "all outgoing ports firewalled except for port 80 and a few other (useless) ones". Not having SSH access is most painful. Luckily, there's a solution.

You need a web server running Apache and SSH. Enable mod_proxy and mod_proxy_connect and add this to the first (i.e. default) virtual host configuration:

<VirtualHost whatever:80>
...

  # allow ssh to localhost over http proxy
  ProxyRequests on
  AllowCONNECT 22
  <Proxy localhost>
    Order allow,deny
    Allow from all
  </Proxy>

</VirtualHost>

Reload Apache configuration. The setup is done. (Instructions based on Tunneling SSH over HTTP(S) by Dag Wieers.)

On the client side you need proxytunnel. Sadly, it's not packaged for Ubuntu yet, but compiling from sources is trivial. Edit ~/.ssh/config and add an entry for your proxied ssh connection:

Host pmyservername
ProxyCommand proxytunnel -q -p myserver.mydomain.com:80 -d localhost:22

That's it. Now you can ssh pmyservername. (The p prefix is a reminder that I'm using a proxied connection: ssh fridge versus ssh pfridge. Also it reminds me of Terry Pratchett's Pyramids.).

For extra fun (e.g. IRC) use ssh's built-in SOCKS5 proxy: ssh -D 1080 pmyservername. Then tell the apps to use a SOCKS5 proxy on localhost. Since telling each app to use a proxy (and then, later, telling it to stop using it) is a big *pain*, and some apps (e.g. ssh) don't support proxies directly, a wrapper like tsocks is handy. Edit /etc/tsocks.conf and set the default socks server to 127.0.0.1, then use it to run apps:

$ tsocks xchat-gnome
$ tsocks bzr push lp:myprojectname

tsocks is packaged for Ubuntu.

If your hotel doesn't have free WiFi, a prepaid SIM card with 3G access could be cheaper than roaming charges. Apparently you can get one with a virtually unlimited (for a short stay, anyway) data plan for 27 EUR in Amsterdam.

I'm at the point in my hobby project where I'd like to be able to change my models without losing all my test data. And I'm too lazy to do manual dumps and edit the SQL in place before reimporting it.

I want a system

• that is transparent to the user: if my database is at schema version 1, and my code is at version 3, I want it to be automatically upgraded to version 3 on server startup.
• that is not too hard on the programmer: dropping a numbered Python or SQL script in a directory ought to be sufficient to define a transition from schema version X to schema version X+1.
• that handles errors gracefully: makes a backup of the database with the old schema version; runs my script in a transaction and aborts that transaction if the conversion fails (while showing me enough information to debug the problem).
• allows prototyping without having to increment the schema number for every little change I make to the models; I should be the one who decides that a new schema is ready to go out to the world.

I've been glancing at SQLAlchemy-Migrate, since I've been brought up to believe NIHing is Bad. But Migrate is scary. I have to admit that the longer I stare at its documentation, the less I can describe why I think so. All those shell commands—but there's an API for invoking them from Python, so maybe I can achieve my goals. I'll have to try and see.

Last time I mentioned that running bin/buildout with the -N flag makes it run faster (since it skips looking for newer versions to upgrade). You can tell buildout to do this by default by putting 'newest = false' into the [buildout] section of buildout.cfg. We'll be running bin/buildout a lot now, since we'll be making changes to the project environment, so this will save wear and tear on the '-', 'N' and Shift keys. (And, by the way, I'm not trying to soak up Google juice by repeating the word 'buildout' a lot, honest!)

I will omit bzr commits from this narrative as it's getting long; you can assume that every self-contained change was committed separately.

tests

First, I want a bin/test script to run the test suite. Pylons uses nose, so we need to tell buildout to install the nosetests script (under a different name, since I'm used to typing bin/test no matter what test runner a project happens to use):

$ bzr diff
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-15 19:49:11 +0000
+++ buildout.cfg	2009-09-15 19:49:18 +0000
@@ -8,5 +8,8 @@
 recipe = zc.recipe.egg
 eggs = Pylons
        PasteScript
+       nose
        asharing
 interpreter = python
+scripts = paster
+          nosetests=test

$ bin/buildout
...
Generated script '/tmp/AlliterationSharing/bin/paster'.
Generated script '/tmp/AlliterationSharing/bin/test'.
...
$ bin/test

----------------------------------------------------------------------
Ran 0 tests in 0.276s

OK

ctags

Documentation is good, but sometimes you want to look at the source code of the framework. There's a tool called ctags that builds a database of identifiers. The popular text editors Vim and Emacs can then use the tags database to jump to a definition of any name with a single keystroke (Ctrl-] in vim, M-. in emacs).

Building the tags database is complicated by each Python package being installed into a separate directory. There's a buildout recipe called z3c.recipe.tag that finds those directories and lets you build a unified tags file. We'll also ask buildout to make sure it unzips any packages distributed as .egg files, since ctags doesn't process those:

$ bzr diff
@@ -1,8 +1,9 @@
 [buildout]
 develop = .
-parts = pylons
+parts = pylons ctags
 
 newest = false
+unzip = true
 
 [pylons]
 recipe = zc.recipe.egg
@@ -13,3 +14,7 @@
 interpreter = python
 scripts = paster
           nosetests=test
+
+[ctags]
+recipe = z3c.recipe.tag:tags
+eggs = ${pylons:eggs}

$ bin/buildout
...
Generated script '/tmp/AlliterationSharing/bin/ctags'.
...
$ bin/ctags

omelette

ctags lets you find classes and functions by name; it doesn't let you find packages or modules. There's another recipe, collective.recipe.omelette that creates a tree of symlinks mirroring the Python package structure (here 'unzip = true' also comes in handy):

$ bzr diff
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-15 20:04:42 +0000
+++ buildout.cfg	2009-09-15 20:05:30 +0000
@@ -1,6 +1,6 @@
 [buildout]
 develop = .
-parts = pylons ctags
+parts = pylons ctags omelette
 
 newest = false
 unzip = true
@@ -18,3 +18,7 @@
 [ctags]
 recipe = z3c.recipe.tag:tags
 eggs = ${pylons:eggs}
+
+[omelette]
+recipe = collective.recipe.omelette
+eggs = ${pylons:eggs}

$ bin/buildout 
...
$ ls -l parts/omelette
...

The symlink tree is created under parts/omelette/. For example, if you want to see what webhelper tags were available, you can open parts/omelette/webhelper/html/builder.py in your editor and see.

Makefile

This is getting long (and not everyone may be interested¹), but one long post is easier to skip than five medium ones in a row, so I'll continue.

¹ Sorry, Planet Maemo! There's an RSS feed of posts tagged 'maemo', if you can figure out the URL, which is very well hidden by PyBlosxom, *sigh*.

Wouldn't it be nice if new developers could check out your project and start it up with just a couple of commands? Make is a time-tested tool that works well for this:

$ cat Makefile
# Just remember that you need to use real tabs, not spaces, in a Makefile

PYTHON = python

.PHONY: all
all: bin/paster

.PHONY: run
run: bin/paster
        bin/paster serve development.ini --reload

.PHONY: test check
test check: bin/test
        bin/test

.PHONY: tags
tags: bin/ctags
        bin/ctags

bin/paster bin/test bin/python bin/ctags: bin/buildout
        bin/buildout

bin/buildout: bootstrap.py
        $(PYTHON) bootstrap.py

Now all you need to do after checking out is run 'make' to set up a working development environment. 'make run' or 'make test' will also do that, if necessary, so this one-liner is sufficient to get a working Hello World application on port 5000:

$ bzr branch lp:~mgedmin/+junk/AlliterationSharing && cd AlliterationSharing && make run

Try it! You'll get a Bazaar branch with all the history of this little blog project.

For software development I prefer buildout to virtualenv. This is because buildout has a text file describing the state of your working environent, which can be versioned and used later to recreate it, as well as during development to modify the environment slightly.

To start a new Pylons project, first create an empty directory. Let's call our new project AlliterationSharing¹, because everybody is sick of 'foo' and 'bar'.

¹ Generated by randomly picking two words from /usr/share/dict/words, then chosen over among 120 other variants that weren't as good.

$ mkdir -p ~/src/AlliterationSharing
$ cd ~/src/AlliterationSharing

Now create a file called buildout.cfg with the following content:

$ cat buildout.cfg
[buildout]
parts = pylons

[pylons]
recipe = zc.recipe.egg
eggs = Pylons
       PasteScript
interpreter = python

Download bootstrap.py to it and run it to get bin/buildout. Note: you can chose which Python version you want to use by running bootstrap.py with it. All other scripts under bin/ will be generated by buildout and will use the same Python interpreter.

$ wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
$ python bootstrap.py
Creating directory '.../AlliterationSharing/bin'.
Creating directory '.../AlliterationSharing/parts'.
Creating directory '.../AlliterationSharing/eggs'.
Creating directory '.../AlliterationSharing/develop-eggs'.
Generated script '.../AlliterationSharing/bin/buildout'.

Run bin/buildout to install Pylons into your sandbox.

$ bin/buildout
Installing pylons.
Generated script '.../AlliterationSharing/bin/paster'.
Generated interpreter '.../AlliterationSharing/bin/python'.

Aside: buildout has this very nice feature where it can share Python packages between projects. This will save you enormous amounts of time that would otherwise be spent downloading and unpacking eggs. To make use of this facility, create a file ~/.buildout/default.cfg with

$ cat ~/.buildout/default.cfg 
[buildout]
eggs-directory = /home/mg/tmp/buildout-eggs
# XXX replace /home/mg with the full path of *your* home directory
# it would be much nicer if buildout let me use ~ or $HOME
# see https://bugs.launchpad.net/zc.buildout/+bug/190260

Another useful trick is to pass the -N flag to bin/buildout, which will tell it not to bother looking for newer versions of packages on the Internet when there's already an existing version installed in your eggs directory.

Back to business: now you've got two new scripts: bin/python and bin/paster. You can use the first one to play with the interactive Python console where you can now import pylons and all the dependencies; it has no other value.

Now is a good point to add the files you've created into a version control system. I'll arbitrarily use Bazaar.

$ bzr init .
$ bzr add bootstrap.py buildout.cfg
$ bzr ignore bin parts eggs develop-eggs .installed.cfg
$ bzr commit -m "Create AlliterationSharing project"

Run bin/paster create -t pylons to create a skeleton project.

$ bin/paster create -t pylons asharing
$ bzr ignore *.egg-info
$ bzr add asharing
$ bzr commit -m "Generated project files with paster create"

Now paster creates a directory structure that I don't like:

AlliterationSharing/
  buildout.cfg
  bin/
  asharing/
    setup.py
    README.txt
    MANIFEST.in
    asharing/
      __init__.py
      config/
      controllers/
      templates/
      public/

I'd like the README and setup.py to be in the top level, and I dislike repeating 'asharing' twice in directory names. I'll move some files around

$ cd asharing/
$ bzr mv development.ini docs MANIFEST.in README.txt setup.* test.ini ../
$ bzr rm ez_setup.*
$ cd ..
$ bzr mv asharing src
$ bzr ci -m "Moved some files around"

Now the tree looks like this:

AlliterationSharing/
  buildout.cfg
  setup.py
  README.txt
  MANIFEST.in
  bin/
  src/
    asharing/
      __init__.py
      config/
      controllers/
      templates/
      public/

We have to tell setup.py where to find the source tree

$ bzr diff
=== modified file 'MANIFEST.in'
--- MANIFEST.in	2009-09-13 13:04:00 +0000
+++ MANIFEST.in	2009-09-13 13:05:59 +0000
@@ -1,3 +1,3 @@
-include asharing/config/deployment.ini_tmpl
-recursive-include asharing/public *
-recursive-include asharing/templates *
+include src/asharing/config/deployment.ini_tmpl
+recursive-include src/asharing/public *
+recursive-include src/asharing/templates *

=== modified file 'setup.py'
--- setup.py	2009-09-13 13:04:00 +0000
+++ setup.py	2009-09-13 13:04:40 +0000
@@ -17,7 +17,8 @@
         "SQLAlchemy>=0.5",
     ],
     setup_requires=["PasteScript>=1.6.3"],
-    packages=find_packages(exclude=['ez_setup']),
+    packages=find_packages('src', exclude=['ez_setup']),
+    package_dir={'': 'src'},
     include_package_data=True,
     test_suite='nose.collector',
     package_data={'asharing': ['i18n/*/LC_MESSAGES/*.mo']},

(I'm not sure if you also need to change package_data and/or setup.cfg; it's possible that I left i18n in a broken state. Can somebody comment on this?)

And we have to tell buildout that we've got a new Python package to enable in the project environment

$ bzr diff buildout.cfg 
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-13 12:57:21 +0000
+++ buildout.cfg	2009-09-13 13:08:05 +0000
@@ -1,8 +1,10 @@
 [buildout]
+develop = .
 parts = pylons
 
 [pylons]
 recipe = zc.recipe.egg
 eggs = Pylons
        PasteScript
+       asharing
 interpreter = python

Now you can re-run bin/buildout and start your hello-world project

$ bzr commit -m "Include the new package in the build"
$ bin/buildout -N
$ bin/paster serve --reload development.ini

Happy hacking!

To be continued: telling buildbot to create bin/test; using ctags and omelette.