Git on Random notes from mg

Converting a gnarly SVN repository to GIT: success!

marius@gedmin.as (Marius Gedminas) — Mon, 20 Feb 2012 23:17:58 +0200

I've received more feedback about my last night's post on gnarly svn to git migration than I've expected. Thanks to that feedback (and, mostly, Raffaele Salmaso for doing almost all the work and emailing the result to me) eazysvn is now on GitHub.

The rest of this post will describe the conversion (and verification) in detail, because if I ever need to do this again, I do not want to start from scratch.

Part one: unexpected gift

Raffaele Salmaso did a heroic job and sent me a tarball with a mercurial repository, produced with "something like" this:

> hg clone --layout single $SVN repo-tmp
> hg convert --filemap filemap repo-tmp repo
> cd repo
> hg qinit
> hg qimport -r 0:tip
> hg qpop -a
> cd .hg/patches
> check patches for correctness
> fix tags (svn are different from mercurial ones)
> hg qfinish -a

The conversion had only two problems:

it introduced new commits that modify a new file .htags
it changed the contents of README.txt in changeset 53 (corresponding to svn revision 55, "Allow branch names to have prefixes.") and newer versions:

--- svn version
+++ hg version
-  *scheme://server/path/to/svn/repo*/*subdirs*
+  *scheme://server/path/to/svn/repo*/trunk/*subdirs*

I did not notice either problem at first.

Part two: conversion to git

Note: I wrote this up after I've done everything up to and including part five. The commands and directory names here are not the actual commands and directories I've used; although I tried to be accurate. I've also skipped some false trails.

Converting hg to git was pretty easy:

$ mkdir -p /tmp/conv
$ cd /tmp/conv
$ tar xvjf eazysvn_20120220-133823.tar.bz2
creates /tmp/conv/eazysvn/
$ mkdir eazysvn-git
$ cd eazysvn-git
$ git init
$ hg-fast-export -r /tmp/conv/eazysvn
converts, leaves no working tree; git status shows all files as deleted
$ git checkout
restore working tree

I was a bit surprised by git status showing a bunch of deleted files at the end there. I suppose hg-fast-export expects to be run inside a bare repository, or maybe it expects the user to know enough git to understand what happened and do the git checkout if necessary.

Part three: cleanup

I wanted to drop the empty changesets that were introduced by Raffaele's conversion process for modifying .hgtags. Since the manual page for git filter-branch had an example for dropping all empty changesets, I used it directly.

$ git filter-branch --commit-filter 'git_commit_non_empty_tree "$@"'

This also dropped some changesets that were present in my Subversion repository -- those that manipulated svn properties, and the one that moved everything in svn root under /trunk. I won't miss those.

Note: if you try

$ git filter-branch --commit-filter='git_commit_non_empty_tree "$@"'

(i.e. '=' instead of a space after --commit-filter), you will get a completely baffling error message that doesn't even hint at what is wrong.

At this point I ran gitk --all to look around and discovered that git filter-branch left all the tags pointing to obsolete revisions. I created new tags manually with gitk, including some that were missing in my svn repository. Every release since 1.6.0 is now tagged (releases before that did not have source tarballs on PyPI, so I had no way to verify which checkin corresponded to which release). I also changed the tag naming scheme to be "v1.x.y" instead of just "1.x.y".

Oh, and to get rid of the old history I did

$ git tag -d 1.9.0 1.11.0 1.12.0 1.12.1
$ git gc --prune

Part four: verification

I downloaded all the available releases from PyPI:

$ mkdir -p /tmp/verify
$ cd /tmp/verify
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   wget http://pypi.python.org/packages/source/e/eazysvn/eazysvn-$v.tar.gz
>   tar xvzf eazysvn-$v.tar.gz
> done

Exported all of my git tags:

$ mkdir -p /tmp/verify/git
$ cd /tmp/conv/eazysvn-git
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   git archive --format=tar --prefix eazysvn-$v/ v$v | gzip \
>       > /tmp/verify/git/eazysvn-$v-git.tar.gz
> done
$ cd /tmp/verify/git
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   tar xvzf eazysvn-$v-git.tar.gz
> done

And compared them:

$ diff -ur /tmp/verify /tmp/verify/git

I expected to see "Only in dir1: setup.cfg" messages only for things like 'eazysvn.egg-info' or 'PKG-INFO'. Unfortunately this is where actual differences I mentioned in part one showed up: in README.txt for all trees starting with release 1.9.0.

Part five: rectification

I needed to rewrite history again, but I didn't want to use git filter-branch this time (I didn't want to manually tag all the releases again). I tried fast-export:

$ cd /tmp/conv/eazysvn-git
$ git fast-export --all > ../EXPORT.txt
$ vim ../EXPORT.txt
search and replace 'repo*/*subdirs*' with 'repo*/trunk/*subdirs*'
fix up the file size above each change (increment by 6, the length of 'trunk/')
$ mkdir /tmp/conv/eazysvn-git2
$ cd /tmp/conv/eazysvn-git2
$ git init
$ git fast-import < ../EXPORT.txt
succeeds, leaves no working tree; git status shows all files as deleted
$ git checkout
restore working tree
$ gitk --all

Everything looked about right.

Part six: final verification

But was it actually right?

$ cd /tmp/verify/
$ mkdir pypi
$ mv eazysvn-*/ pypi/
$ rm -rf git/*
$ cd /tmp/conv/eazysvn-git2
$ for v in 1.6.0 1.6.1 1.7.0 1.8.0 1.9.0 1.10.0 1.11.0 1.12.0 1.12.1; do
>   git archive --format=tar --prefix eazysvn-$v/ v$v \
>       | (cd /tmp/verify/git && tar -xf - )
> done
$ cd /tmp/verify
$ diff -ur pypi git

Yes!

Part seven: uploading to GitHub

It was all plain sailing from here:

$ cd /tmp/conv/eazysvn-git2
$ git remote add origin git@github.com:mgedmin/eazysvn.git
$ git push -u origin master
$ git push --tags

And then there were documentation updates (to point to GitHub instead of the old Subversion repository), Makefile updates (make release makes sure my sdist contains everything I have in my repository, because I've been bitten by setuptools magic before), etc.

I also released eazysvn 1.12.2 to PyPI to test my Makefile changes, and because there were unreleased changes that should've been released a long time ago.

So that's it. Only took me three hours from the point where I found a Mercurial repository in my inbox.

Converting a gnarly SVN repository to GIT: FAIL.

marius@gedmin.as (Marius Gedminas) — Mon, 20 Feb 2012 00:13:44 +0200

eazysvn lives in a Subversion repository. I want to bring it (kicking and screaming) into the 21st century and put it on Github.

git-svn is unsuitable for the conversion, because in revision 50 I moved / to /trunk and added the traditional /tags and /branches. With git-svn I either get one third of the history that ignores everything before the layout switch, or I get directories named 'trunk', 'tags' and 'branches'.

Then I thought maybe hg would be smarter about the conversion, and then I could use hg-fast-export to convert hg to git. I enabled the hgsubversion extension:

$ sudo apt-get install hgsubversion
$ echo '[extensions'] >> ~/.hgrc
$ echo 'hgsubversion =' >> ~/.hgrc

(blog posts like this are a good reason why hg ought to steal the 'git config --global foo.bar=baz' syntax from git).

Then I converted the svn repository to Mercurial:

$ hg clone svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-hg

hg log -p ~~confirmed that hg handled the conversion nicely~~ looked right-ish at first glance, except the author information was nonsensical. To fix that:

$ rm -rf eazysvn-hg
$ echo 'mg = Marius Gedminas <marius@gedmin.as>' > AUTHORS
$ hg clone svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-hg -A AUTHORS

Unfortunately, a closer look at hg log now shows that two thirds of the history is lost: hg ignored everything before the layout restructuring, despite printing the log messages of those revisions as it went about the conversion. *sigh*.

Dear lazyweb, surely svn layout reorganization can't be such a rare thing that no tools in existence support it? What should I try next?

P.S. I also tried Bazaar, to see what it would do:

$ bzr branch svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-bzr
Repository with UUID 4fc293c4-4eed-0310-a01a-b4ad72f90fad at svn+ssh://fridge/home/mg/svn/eazysvn contains fewer revisions than cache. This either means that this repository contains an out of date mirror of another repository (harmless), or that the UUID is being used for two different Subversion repositories (potential repository corruption).
bzr: ERROR: exceptions.KeyError: 'missing revision paths for 78'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 946, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1150, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 699, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 721, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1263, in run
    from_location)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 919, in open_tree_or_branch
    return bzrdir._get_tree_branch()
  File "/usr/lib/python2.7/dist-packages/bzrlib/controldir.py", line 410, in _get_tree_branch
    branch = self.open_branch(name=name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/remote.py", line 420, in open_branch
    branch_path = self._determine_relpath(name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/remote.py", line 369, in _determine_relpath
    layout = repos.get_layout()
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 701, in get_layout
    return self.get_layout_source()[0]
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 720, in get_layout_source
    self._find_guessed_layout(self.get_config())
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 743, in _find_guessed_layout
    revnum, self._hinted_branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 143, in repository_guess_layout
    return logwalker_guess_layout(repository._log, revnum, branch_path=branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 149, in logwalker_guess_layout
    logwalker.iter_changes(None, revnum, max(0, revnum-GUESS_SAMPLE_SIZE)), revnum, branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 104, in guess_layout_from_history
    for (revpaths, revnum, revprops) in changed_paths:
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/logwalker.py", line 60, in iter_all_changes
    revpaths = get_revision_paths(revnum)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/logwalker.py", line 295, in get_revision_paths
    return self.cache.get_revision_paths(revnum)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/cache/tdbcache.py", line 187, in get_revision_paths
    raise KeyError("missing revision paths for %d" % revnum)
KeyError: 'missing revision paths for 78'

You can report this problem to Bazaar's developers by running
    apport-bug /var/crash/bzr.1000.2012-02-19T22:12.crash
if a bug-reporting window does not automatically appear.