Frozen repositories

OLPC runs a frozen repositories server at mock.laptop.org. This is where we clone Fedora and OLPC packages (source and binary) that are shipped in official OLPC Releases. When OLPC or deployments use the build system to construct/reconstruct official releases, packages are downloaded from this server.

According to Michael Stone, the system was designed with these goals:

1. To help OLPC meet its GPL obligations to offer source code for the binaries it has distributed
2. To lessen the risk of schedule slippage due to Fedora downtime (e.g. resulting from security incidents or hardware failures)
3. To establish other conditions necessary for build reproducibility (like knowing what packages went into each build)
4. To lessen the risk of undetected build data corruption (at rest or in transit) due to Murphy

This page is mostly for those interested or involved in OLPC's software Release Process.

This server is maintained by User:DanielDrake and Chris Ball. Questions and queries go to the devel mailing list.

How it works

A series of repositories are viewable at http://mock.laptop.org/repos. A given release pulls from a specific set of these repositories.

Each repository is a git tree, which can be cloned over the git:// protocol. Some HTTP rewrite rules also expose the HEAD of each repository as regular files (this allows you or the build system to reference a URL such as http://mock.laptop.org/repos/koji.dist-f11 as a yum repository and it will "just work").

The repositories prefixed with "koji." indicate that they are clones at specific points in time of repositories from Fedora's repository server at http://koji.fedoraproject.org.

The repositories prefixed with "local." indicate packages that OLPC add (or override Fedora's versions).

Administration

Creating a new, empty repository

In any of the following instructions, when you are asked to create a repository, this is the case you go through:

(while logged into mock.laptop.org)
cd /srv/git
mkdir my-new-repo
cd my-new-repo
git --bare init --shared
git config core.compression 0
git config pack.window 0
git config pack.packSizeLimit 10m
git config pack.windowMemory 10m
git config pack.depth 1
git config pack.deltaCacheSize 1
git config pack.deltaCacheLimit 1
git config core.bigFileThreshold 1

Creating a repository based on an existing repo

In the cases where one repository logically builds on from one that came before it (e.g. local.11.2.1 is just local.11.2.0 with a few changes on top), you should create the new repository initially as a clone of the first one. This way, git will use hardlinks to the object packs and there will be no duplication of data or wastage of disk space.

cd /srv/git
git clone --bare old-repo new-repo
find new-repo -type f -exec chmod 664 {} +
find new-repo -type d -exec chmod 2775 {} +
cd new-repo
git config core.compression 0
git config pack.window 0
git config pack.packSizeLimit 10m
git config pack.windowMemory 10m
git config pack.depth 1
git config pack.deltaCacheSize 1
git config pack.deltaCacheLimit 1
git config core.bigFileThreshold 1

Checking out a repository to modify it

When you want to make a change to a repository, from your home directory you can work as follows:

git clone /srv/git/the-repo
cd the-repo
git config core.compression 0
git config pack.window 0
git config pack.packSizeLimit 10m
git config pack.windowMemory 10m
git config pack.depth 1
git config pack.deltaCacheSize 1
git config pack.deltaCacheLimit 1
(make changes here)
git commit -a -q
git push

Always use the "-q" option to make commits. Otherwise, git will try and produce a pointless diff, this can take a lot of time.

When you clone like this, the the-repo/.git directory will take up almost zero space, because it just creates hardlinks to the "real" git objects in /var/git. However, the accompanying files checkout in the-repo/ will eat disk space. So please remember to clean up your home directory once you are done with a release.

Creating a new Fedora repository

If producing a software release based on a Fedora version that has not previously been used in an official release, you will need to clone the Fedora "release" repository of the corresponding version onto the frozen repositories server.

Follow the above instructions to create a new, empty repository, and to check it out in your home directory. The repository should be named according to the following scheme: koji.dist-f<FEDORA_RELEASE>-<ARCHITECTURE> e.g. dist-f14-i686.

Now, inside your git checkout, rsync in all the Fedora RPMs and SRPMs of that release, e.g. for Fedora 14:

# mkdir -p RPMS SRPMS
# rsync -av --progress --delete rsync://mirrors.rit.edu/fedora-enchilada/linux/releases/14/Everything/i386/os/Packages/ RPMS/
# rsync -av --progress --delete --exclude repodata rsync://mirrors.rit.edu/fedora-enchilada/linux/releases/14/Everything/source/SRPMS/ SRPMS/

Next, we must download the comps.xml file. Find it e.g. here (F14) and save it as comps.xml in the root of your git checkout.

Next, create the yum database with:

# createrepo --simple-md-filenames -g comps.xml -d .

Finally, commit it all to git and push it:

# git add .
# git commit -q
# git push

Creating a new Fedora updates repository

Each release will pull from its own fedora-updates repository. We form this from a combination of Fedora's latest official updates repository at the time when we start using mock to produce release candidates for a specific release, plus (usually) a few packages pulled in from updates-testing.

Repositories should be named as follows: koji.dist-f<FEDORA_RELEASE>-<ARCHITECTURE>-updates-<OLPC_RELEASE> e.g. koji.dist-f14-i686-updates-11.2.0.

If there is already a repository of Fedora updates for the same architecture and Fedora release that you are building for, use this as a base to create the new repository. Otherwise, if there is no existing repository of Fedora updates of the same architecture and Fedora release, start from an empty repo. Create a checkout in your home directory.

Now follow these instructions (which work both for new repositories and those cloned from earlier ones):

# mkdir -p RPMS SRPMS
# rsync -av --progress --delete --exclude debug --exclude repodata --exclude drpms rsync://mirrors.rit.edu/fedora-enchilada/linux/updates/14/i386/ RPMS/
# rsync -av --progress --delete --exclude repodata rsync://mirrors.rit.edu/fedora-enchilada/linux/updates/14/SRPMS/ SRPMS/

Now download new comps.xml manually from e.g. here, for F14. Create the repodata, commit, and push:

# createrepo --simple-md-filenames -g comps.xml -d .
# git add .
# git commit -a -q
# git push

If you are following these instructions, it is likely because you are taking a development release into stabilization stages - i.e. you are switching from using live Fedora repositories to frozen OLPC repositories. If you do a build from your new frozen repositories, however, some packages won't match up. This is because development builds tend to include updates-testing, and we only cloned the "updates" repository.

The normal process to work around this is to simply add the relevant updates from updates-testing into the repository you have just created. This can be done by creating a local build based on the new repository, then looking at a diff of the packages.txt between the last released development build and your local test build to identify which packages get downgraded when switching to the frozen repos. Normally about 10 packages are affected.

Then, for each package you identify:

1. Identify the update on https://admin.fedoraproject.org/updates/
2. For each update, follow the links to all the koji builds included in that update
3. For each koji build, download all the RPMs and SRPMs, putting them under RPMS/ and SRPMS/ respectively, deleting any old versions.
4. Put a bodhi comment on each update stating that the update has been tested in OLPC builds without any reported problems.

Finally, regenerate the metadata, and commit and push your changes:

# createrepo --simple-md-filenames -g comps.xml -d .
# git commit -a -q
# git push

Creating a new OLPC-local repository

A release will pull from a few OLPC-local repositories. These are created individually, with a one-to-one correspondence to RPM dropboxes that were used in development builds.

The naming scheme is best explained by examples: local.11.2.0, local.11.2.0-xo1.5, local.11.2.0-xo1

The following process should be followed for each dropbox that you wish to convert to a frozen repo.

If the repository is for a brand new major release, create an empty repository. Otherwise, for minor releases, create a repository based on the release that came before it, e.g. local.11.2.1 should start as a clone as local.11.2.0. Check this out in your home directory.

Access the RPM dropbox and examine the packages that it includes. Using your knowledge and referencing packages.txt of the latest development build, take the latest version of each package in the dropbox that is needed for build inclusion. Put its RPM under RPMS/, and its SRPM under SRPMS/. Remove older versions of that package, if there are any.

Finally, create metadata, commit, and push:

# createrepo --simple-md-filenames -d .
# git add .
# git commit -a -q
# git push

Adding/updating packages

Frozen repositories are created at the point where a release goes into release candidate stages. Usually, a few package updates will be needed from that point, forming further release candidates, before the latest candidate is marked as the official release.

First, you must decide which repository the package belongs in. Follow this logic:

• If the package is already in a OLPC-local repository, the update belongs in that OLPC-local repository
• If the package update comes from fedora-updates or fedora-updates-testing, the update belongs in the relevant koji.dist-fXX-updates- repository.
• Otherwise, hopefully in rare cases only, the package should be added to a OLPC-local repository, based on either its generic application or whether it is laptop-specific.

After identifying which repository the package belongs in, put it there, additionally adding the SRPM. Delete old versions of the same package. Then regenerate metadata, commit and push:

# createrepo --simple-md-filenames -g comps.xml -d .
# git add .
# git commit -a -q
# git push

For local repositories, the "-g comps.xml" argument of the createrepo command must be dropped.

Finalizing a release

After a release candidate has been marked as a final release, log into mock.laptop.org and perform the following tasks:

1. Clean up the repository checkouts in your home directory. The release is done, you won't be using them any more.
2. Mark all the repositories used by the release as read-only, as a safeguard against their future modification:

chmod -R a-w /srv/git/repo1 /srv/git/repo2

Previous system design

An earlier iteration of the system had all of the repositories as (mostly unrelated) branches of a single monster-repo. This got huge (over 100GB) and git didn't really like it. You'll find no shortage of discussions resulting in the fact that git is simply not designed for large collections of binary files. We now split the repositories into physical separate git repositories so that each one is short lived (only during its corresponding release cycle).

This is why some of the older git repositories have strange looking history - for example, koji.dist-f11 can be traced back through Fedora 9 and then Fedora 7 -- with those distro changes happening at points where the entire contents of the tree are erased and then replaced with the new distro packages.

To convert to the new system where each branch is its own repository, the history of each branch was examined. If history of a branch was shared with a branch that had already been split off, the repository for the new branch was created based on a local clone (sharing hardlinks to pack objects) of the other to save disk space, then the new branch was pushed to that cloned repository. This space saving might be undone if someone unknowingly comes along and does "git repack" or something, but those repositories are now read-only and not expected to be touched in future.