Random notes from mg

a blog by Marius Gedminas

Marius is a Python hacker. He works for Programmers of Vilnius, a small Python/Zope 3 startup. He has a personal home page at http://gedmin.as. His email is marius@gedmin.as. He does not like spam, but is not afraid of it.

Mon, 21 Sep 2009

Pylons and SQL schema migration

I'm at the point in my hobby project where I'd like to be able to change my models without losing all my test data. And I'm too lazy to do manual dumps and edit the SQL in place before reimporting it.

I want a system

I've been glancing at SQLAlchemy-Migrate, since I've been brought up to believe NIHing is Bad. But Migrate is scary. I have to admit that the longer I stare at its documentation, the less I can describe why I think so. All those shell commands—but there's an API for invoking them from Python, so maybe I can achieve my goals. I'll have to try and see.

posted at 19:44 | tags: , , | permanent link to this entry | 9 comments
Yeah, I have the same opinion about migrate, and rolled my own system for Ibid. I very quickly came across the lack of cross-DB consistency in SQLAlchemy, and general lack of DDL features. You can see the horrific result here. I get the feeling I need to completely re-invent the DDL side, or use migrate
posted by Stefano Rivera at Mon Sep 21 21:43:53 2009
Stefano: interesting approach!

I spent three hours playing with SQLAlchemy-migrate, and I'm rather unhappy with it.  No transparency, no graceful error handling, and, worst of all, showstopper bugs that, e.g., prevent me from dropping a column in a table that has a (different) column called 'order' in it.

You scare me by complaining about raw SA, since that's what I was considering now.

Somebody on IRC mentioned miruku as a possible alternative, but given its low version number (0.1.1) and the broken egg on PyPI (both buildout and easy_install fail to install it), I'm not planning to investigate it.
posted by Marius Gedminas at Mon Sep 21 22:56:29 2009
The problem with SQLAlchemy is that it just doesn't give you abstractions for this stuff, so you have to make them yourself. Without modifying SQLAlchemy, you have to do some nasty hackery. SQLAlchemy 0.6 is supposed to include more DDL commands, but we'll have to see.

The bug you linked to can easily be fixed, by using SQLAlchemy's DBMS-language modules.

The biggest issue I've been having with SQLAlchemy is that it doesn't have exactly the same results on different DBMSs. I.e. index creating semantics on MySQL are different to Postgres, and it doesn't abstract that away. If your code is only going to be used on one system, doing this in SQLAlchemy isn't such a bad solution any more.
posted by Stefano Rivera at Tue Sep 22 00:15:55 2009
Miruku doesn't look like it does much, but then I haven't read the source...
posted by Stefano Rivera at Tue Sep 22 00:16:35 2009
Stefano -

I see a lot of things in your sample code which are easily solved if you just submit patches to SQLA rather than reinvent.  Such as the parsing of the SQL error after saying "index.create()" - we have ticket #527 which addresses supporting the "createfirst" functionality for index.create(), and the patch to provide that functionality is very straightforward - it just requires adding a has_index() method to each dialect.  Similarly the handling of ALTER COLUMN in MySQL to handle its quirks regarding TEXT, just a simple ticket + patch to the migrate project.

I see a lot of other weird comments that would be better addressed if you wrote to the mailing list, like "SQLAlchemy indexes aren't attached to tables" (they are), "SQLAlchemy doesn't pick up all the indexes in the reflected table" (this is a feature that is now available and we would welcome further test cases if you have some which fail), "SQLIte doesn't like unique but not indexed columns" (define "doesn't like"?  if a compelling case can be made that unique is pointless without index, it can be added). 

SQLAlchemy really would have no capabilities at all if we didn't have users using the project, reporting bugs, submitting patches, etc. 

Also the issue of Migrate error handling is a difficult one to solve - while you can run ALTER statements on Postgresql inside of a transaction, and therefore roll them back when something goes wrong, most databases don't offer that capability - ALTER and CREATE statements commit at once.  Hence they didn't build for it.  It would be nice if they supported tranactional scripts, and in fact I've monkeypatched my own to provide this, but then again that would be another "not cross compatible" feature.
posted by mike bayer at Tue Sep 22 03:51:03 2009
Also regarding 0.6, this is not in "we'll see" status - the current 0.6 trunk is already used in some production applications.  The docs are up and I've written some new documentation on the DDL compilation support, with some more to go.  I've also submitted patches to Migrate such that they can begin building off of SQLA's DDL constructs some more, which would simplify Migrate and also move a bigger proportion of its per-database support over to SQLA.  This also should allow other migrate systems to be easier to build off of SQLA.  The custom DDL sequences stuff is described at http://www.sqlalchemy.org/docs/06/metadata.html#customizing-ddl .
posted by mike bayer at Tue Sep 22 03:55:30 2009
sorry, one more link, if you'd like to build constructs like AlterTable and such which compile to different backends, the compiler extension is what you'd use:  http://www.sqlalchemy.org/docs/06/reference/ext/compiler.html#synopsis
posted by mike bayer at Tue Sep 22 03:59:00 2009
Having a proper tool for migrations would be very useful. Ruby on Rails has it's migrations which work nice, but there is nothing really useable on the Python side.
posted by Tõnis at Tue Sep 22 09:26:44 2009
mike: Yes, you are right - I should play with SA 0.6, and report those bugs upstream. That code was mostly written experimentally - I don't want to have to maintain those hacks, rather get fixes upstream.

However, we also want to support SA>=0.4 (i.e.  versions in ubuntu LTS and debian stable), so we'll have to carry the hacks for a while
posted by Stefano Rivera at Tue Sep 22 15:27:53 2009

Name (required)


E-mail (will not be shown)


URL


Comment (some HTML allowed)