Frozen repositories

OLPC runs a frozen repositories server at mock.laptop.org. This is where we clone Fedora and OLPC packages (source and binary) that are shipped in official OLPC Releases. When OLPC or deployments use the build system to construct/reconstruct official releases, packages are downloaded from this server.

According to Michael Stone, the system was designed with these goals:

1. To help OLPC meet its GPL obligations to offer source code for the binaries it has distributed
2. To lessen the risk of schedule slippage due to Fedora downtime (e.g. resulting from security incidents or hardware failures)
3. To establish other conditions necessary for build reproducibility (like knowing what packages went into each build)
4. To lessen the risk of undetected build data corruption (at rest or in transit) due to Murphy

This page is mostly for those interested or involved in OLPC's software Release Process.

This server is maintained by User:DanielDrake and Chris Ball. Questions and queries go to the devel mailing list.

How it works

A series of repositories are viewable at http://mock.laptop.org/repos. A given release pulls from a specific set of these repositories.

Each repository is a git tree, which can be cloned over the git:// protocol. Some HTTP rewrite rules also expose the HEAD of each repository as regular files (this allows you or the build system to reference a URL such as http://mock.laptop.org/repos/koji.dist-f11 as a yum repository and it will "just work").

The repositories prefixed with "koji." indicate that they are clones at specific points in time of repositories from Fedora's repository server at http://koji.fedoraproject.org.

The repositories prefixed with "local." indicate packages that OLPC add (or override Fedora's versions).

Administration

Creating a new, empty repository

In any of the following instructions, when you are asked to create a repository, this is the case you go through:

(while logged into mock.laptop.org)
cd /var/lib/git
mkdir my-new-repo
cd my-new-repo
git --bare init --shared
git config core.compression 0
git config pack.window 0
git config pack.packSizeLimit 400m

New repositories may take up to an hour to be accessible over http.

Checking out a repository to modify it

When you want to make a change to a repository, from your home directory you can work as follows:

git clone /var/lib/git/the-repo
cd the-repo
(make changes here)
git commit -a -q
git push

Always use the "-q" option to make commits. Otherwise, git will try and produce a pointless diff, this can take a lot of time.

When you clone like this, the the-repo/.git directory will take up almost zero space, because it just creates hardlinks to the "real" git objects in /var/git. However, the accompanying files checkout in the-repo/ will eat disk space. So please remember to clean up your home directory once you are done with a release.

Creating a new Fedora repository

If producing a software release based on a Fedora version that has not previously been used in an official release, you will need to clone the Fedora "release" repository of the corresponding version onto the frozen repositories server.

Follow the above instructions to create a new, empty repository, and to check it out in your home directory. The repository should be named according to the following scheme: koji.dist-f<FEDORA_RELEASE>-<ARCHITECTURE> e.g. dist-f14-i686.

Now, inside your git checkout, rsync in all the Fedora RPMs and SRPMs of that release, e.g. for Fedora 14:

# mkdir -p RPMS SRPMS
# rsync -av --progress --delete rsync://mirrors.rit.edu/fedora-enchilada/linux/releases/14/Everything/i386/os/Packages/ RPMS/
# rsync -av --progress --delete --exclude repodata rsync://mirrors.rit.edu/fedora-enchilada/linux/releases/14/Everything/source/SRPMS/ SRPMS/

Next, we must download the comps.xml file. Find it here and save it as comps.xml in the root of your git checkout.

Next, create the yum database with:

# createrepo -g comps.xml -d .

Finally, commit it all to git and push it:

# git add .
# git commit -q
# git push

Finalizing a release

After a release candidate has been marked as a final release, log into mock.laptop.org and perform the following tasks:

1. Clean up the repository checkouts in your home directory. The release is done, you won't be using them any more.
2. Mark all the repositories used by the release as read-only, as a safeguard against their future modification:

chmod -R a-w /var/lib/git/repo1 /var/lib/git/repo2

Previous system design

An earlier iteration of the system had all of the repositories as (mostly unrelated) branches of a single monster-repo. This got huge (over 100GB) and git didn't really like it. You'll find no shortage of discussions resulting in the fact that git is simply not designed for large collections of binary files. We now split the repositories into physical separate git repositories so that each one is short lived (only during its corresponding release cycle).

This is why some of the older git repositories have strange looking history - for example, koji.dist-f11 can be traced back through Fedora 9 and then Fedora 7 -- with those distro changes happening at points where the entire contents of the tree are erased and then replaced with the new distro packages.

To convert to the new system where each branch is its own repository, the history of each branch was examined. If history of a branch was shared with a branch that had already been split off, the repository for the new branch was created based on a local clone (sharing hardlinks to pack objects) of the other to save disk space, then the new branch was pushed to that cloned repository. This space saving might be undone if someone unknowingly comes along and does "git repack" or something, but those repositories are now read-only and not expected to be touched in future.