Moving from Subversion to Kiln with Mercurial

The Kiln Importer is a great tool, but it’s also a bit simple: put in a repository on one end and get a repository in Kiln on the other end. Tweaking your repository so that it’s ideally set up for Mercurial workflows is left as an exercise to the user.

While we’re planning to make changes to the Kiln Importer to make it more advanced, here are some principles you can apply to manually convert your repository so that it works well in Mercurial.

Convert only what you need

Some version control systems, such as Subversion, will structure the repository as a file tree such that subdirectories are used to represent individual projects. In Kiln and Mercurial, you most likely do not want to convert the entire root trunk of the source repository.

A good test for determining which paths should be independent mercurial repositories is to ask, “which path would I use to checkout code if I needed to begin work on a single project and only this project?” That’s usually the path you want to use when converting to Mercurial.

For example, let’s say your repository tree in SVN looks like this:

/Trunk
    /CoolProject
         /WebApp
         /DesktopApp
    /Utilities
         /userscripts
         /powershell

In the above example, you would probably want to have Mercurial repositories for the following paths:

/Trunk/CoolProject/WebApp
/Trunk/CoolProject/DesktopApp
/Trunk/Utilities

Although CoolProject/WebApp and CoolProject/DesktopApp are related, they probably should be independent repositories unless you always work on development together. This also helps to separate the history of WebApp and DesktopApp, which are distinct products.

We could separate the Utilities/userscripts and Utilities/powershell repositories since they may not be related, it may simply be more convenient to work on all utilities at once. Further, I’m not concerned if the history of userscripts gets mixed with the history of Powershell.

Use the hgconvert extension

The Kiln import tool will do a good job converting your entire repo, but sometimes you want to be a bit more careful with what you bring over. For example, if you had binary debugging files that you would like to exclude from Mercurial’s history, you may want to do the conversion manually.

Start with the documentation for the hgconvert extension. You’ll need to enable the extension before using it. You can enable it by adding the following lines to your Mercurial.ini file:

[extensions]
hgext.convert=

You can use hg convert to convert from many different sources in very precise ways. You can even use hg convert against other Mercurial repositories.

Use hgconvert filemaps to exclude files

While you can use an .hgignore file to exclude certain files from your repository (see below), it’s not going to help remove files from your repository’s history. Do remove files and directories from the entire history of your repository, you need to use hg convert --filemap.

The documentation for –filemap is a bit sparse, so I’m going to include a few additional details here.

The filemap is a text file that you’ll reference during the conversion. It allows you to specify multiple lines with exclude, include, or rename directives. Note that include will only include the files that are specified in include and will automatically exclude anything not included in an include statement. If you’re using filemap to replicate .hgignore, you’ll probably stick with exclude.

Let’s say you create a filemap at C:\Temp\filemap.txt with the following contents:

exclude "Test.txt"     # exclude the file Test.txt from the 
                       # converted repo
exclude "libs/abc.dll"# use unix-style paths, regardless of OS
exclude "subdir"       # exclude all files from this subdirectory

In the example above, the first line will exclude Test.txt. Note that filenames are case sensitive. The second line will exclude abc.dll, which is in the subdirectory libs. Note the use of unix-style paths, which is required. The third line will exclude the entire subdirectory called subdir.

When I’m ready to run hgconvert, I can do so with a command like the following:

> hg convert --filemap C:\Temp\filemap.txt 
    http://path/of/source/repo .\DestinationPath

After running hg convert, you will have a new repository that excludes the files you specified from its entire history.

Use kbfiles and kbfconvert (largefiles as of Kiln 2.9)

NOTE: Kiln 2.9 ships with support for the LargeFiles extension, which is now part of Mercurial and not exclusive to Kiln. Most of the following still applies, but you should use LargeFiles instead of KBFiles. See How do I use the Mercurial LargeFiles Extension? for more details.

Kiln ships with an extension called kbfiles which can be useful for storing large binary assets in your repository. If you have large files in your repository for which having an atomic diff is unimportant, it’s recommended (and sometimes required) that you use kbfiles.

For example, let’s say you store documentation.pdf, a 35MB binary file, in your repository. Each time you commit a change to documentation.pdf, unless you have added the file as a bfile, Mercurial is going to diff the file to try to figure out exactly what changed. This is very resource intensive on Mercurial, and very often unnecessary. Instead, we use kbfiles, which will still store each revision of the file, but it won’t try to diff it because a diff would be meaningless.

See How do you use kbfiles? for how to use kbfiles and additional explanation of why you want to use it. That documentation will give you a good idea of how to add kbfiles to an existing repository. I want to focus here on the conversion features of kbfiles with kbfconvert.

You can use kbfconvert to convert your repository to one with a new history so that specific files or files of a particular size are always added as kbfiles for the entire history of the repository. This is preferred if you’re going to be pushing large binary assets to Kiln.

If you want to convert a repository to use kbfiles purely based on size, you can use a command like the following, which will automatically convert all files larger than 10 MB to use kbfiles:

> hg kbfconvert sourcerepo destrepo --size 10

What if you want to add files not based on size, but explicitly by file name? You can use kbfconvert to specify the names of individual files to add as kbfiles. The files you specify get added even if they are under the size specification. In the following example, TestSearch3.png, Test.txt.orig, and subdir\file2.txt are all going to become kbfiles in the converted repository, even though they are smaller than the specified file size of 1000MB.

> hg kbfconvert .\SourceRepo.\DestinationRepo TestSearch3.png 
    Test.txt.orig subdir\file2.txt --size 1000

Use a command like the above to explicitly convert specific files to using kbfiles in the converted repository.

Use an .hgignore file

Most Mercurial repositories benefit from an .hgignore file because it helps remove unnecessary cruft from the repository. For example, a Visual Studio solution is going to include .pdb debugging files and compiled binaries that don’t need be included in the repository. In this case, an .hgignore is essential.

The documentation for hgignore is a good start. You might be able to search around for a sample .hgignore that matches the development environment for the project you’re working on, but be careful that it’s not too liberal in excluding files. For example, this one will match a file like “/project/Debugger/something.cs”, which is probably undesirable. You can add the .hgignore file after the conversion, but it is not used by the hg convert process (unless of course you’re converting a Mercurial repository that included an .hgignore file in its history).

Bringing it all together

We’ve gone over quite a few techniques above. Here’s a set of practical steps you can apply to convert your repositories so that they’re in good shape for use in Kiln.

Identify which paths in your source version control system are going to be converted into individual mercurial repositories. You’ll want to run the following steps for each repository.
Create a filemap for the repository so that you exclude files that do not need to be in the converted repository.
Run hg convert with the filemap you created so that you now have a Mercurial repository with only the files you need to actually work on code.
Run kbfconvert against the converted repository from step 3 such that some files, specified either explicitly or by size, are included as kbfiles. The resulting repository will be much friendlier to Kiln and Mercurial.
Add an .hgignore file to the converted repository from step 4 so that going forward, the repository only contains what it needs. Don’t forget to hg add and hg commit the .hgignore file!
Push the repository to Kiln.

Once you do the above steps, you should be good to go!