How to Use the Mercurial LargeFiles Extension

Should I use LargeFiles at all?

Probably not. The benefit of LargeFiles is to save some bandwidth on push/pull/clone commands and save a few CPU cycles when making a commit. This comes at a cost though! LargeFiles is considered a “last resort” extension for Mercurial because it breaks the D in DVCS (“Distributed Version Control System”).

There is a very specific problem that LargeFiles is designed to solve: in the case where a repository has binary files which change frequently, the changesets can get very large because binary files are not very easy to diff or to compress. LargeFiles basically solves this problem by allowing certain files to be treated in the same way that Source Control Systems like Subversion treated them: as snapshots of the file at certain times. Then when you do a pull, clone, or update command, you only get the snapshots of the LargeFiles that correspond with the relevant revision. This means you do not have a full copy of the repository on your local machine. You rely on the central repository (on Kiln) to store all the revisions of the LargeFiles and only pull them over the network when you update to a revision that needs them. This saves bandwidth and some CPU cycles.

I understand the cost/benefits trade-off, now how do I use LargeFiles?

Great! Are you sure? Go read the previous section one more time and be sure you really need LargeFiles. I’ll wait… Okay, you seem pretty sure, so let’s proceed.

To use LargeFiles you just need to enable the extension and start adding files with the ‘–large’ parameter like ‘hg add –large largefile.pdf’. You might already have a bunch of binary files that you want to convert to be large files instead, which means you’ll have to convert the repository.

NOTE: Once you convert your repository to use LargeFiles, it will no longer be related to the original repository. Because of this, you’ll want to choose to do the conversion at a point in time when your entire team can switch to using the newly-converted LargeFiles repo.

Turning on the LargeFiles extension

Neither Kiln nor Mercurial will enable this extension by default when you install them. In order to turn the extension on you will need to add the following lines to the extensions section of your mercurial configuration file:

[extensions]
largefiles =

Do I have to convert my existing KBFiles-enabled repositories?

No, you don’t have to convert your repositories. KBFiles is still working with Kiln 2.9. However, LargeFiles is the successor to KBFiles so there will be no more updates or bug-fixes for KBFiles. Additionally, LargeFiles resolves an issue that existed in KBFiles which reduced the performance of many hg commands. So, it is not strictly required but it is highly recommended that you switch to LargeFiles to avoid the performance impact of KBFiles and to receive any code updates to the extension.

How do I perform the conversion from KBFiles to LargeFiles?

First, you’ll need to be sure you have the latest version of the Kiln Client and Tools. You can download this from the Kiln page under “Resources”. Once you have this installed, the conversion is a simple two-step process. Both KBFiles and LargeFiles have methods to convert from a normal repository to one using the extension and vice-versa, so it is through these commands that we will perform the conversion.

You need to first convert from a KBFiles-enabled repository to a normal one, and then convert that normal repository to a LargeFiles-enabled one. The following two commands will perform these conversions (the ‘–size 10’ parameter in the second command indicates that any file over 10MB should be converted to a LargeFile. See the LargeFiles page for more complex patterns):

hg kbfconvert --tonormal /your/kbf/repo/location /temporary/normal/repo
hg lfconvert --size 10 /temporary/normal/repo /new/largefiles/repo

Now you have a new LargeFiles repository with all the same data as was in your original repository! You can push this new repository into Kiln and get everyone to start using it. The temporary repository created by the first command can safely be deleted from your local disk, and it would probably be wise to delete the original KBFiles-enabled repository from Kiln to ensure that everyone switches to using the new LargeFiles repository (don’t worry, kiln doesn’t delete the data from the repository, it just marks it as deleted and makes it inaccessible to non-admins. You can recover it later if anything goes wrong).

How do I enable LargeFiles on a normal repository?

First, you’ll need to be sure you have the latest version of the Kiln Client and Tools. You can download this from the Kiln page under “Resources”. Then you just need to convert the repository with the following command (the ‘–size 10’ parameter indicates that any file over 10MB should be converted to a LargeFile. See the LargeFiles page for more complex patterns):

hg lfconvert --size 10 /temporary/normal/repo /new/largefiles/repo

Now push this new repository into Kiln and start using it! You can delete the original normal repository from Kiln to ensure that everyone switches to using the new LargeFiles repository (don’t worry, kiln doesn’t delete the data from the repository, it just marks it as deleted and makes it inaccessible to non-admins. You can recover it later if anything goes wrong).