User:DanielDrake/NewMockProposal

THIS IS A PROPOSAL IN DRAFT STATE

mock.laptop.org is our "frozen repository" host.

The normal OLPC OS development process starts with RPMs coming in from Fedora's core, updates, and updates-testing repositories. There is no control on which packages go in and out, and this is only loosely monitored. Each development build simply takes the latest that Fedora is offering.

Later in the development cycle, OLPC decides to freeze the packages going into the build, to provide a more suitable environment for testing. At this point, an OLPC developer clones the packages being used onto mock.laptop.org in a manner which is mostly undocumented, and then all development builds use mock.laptop.org rather than Fedora as their source of packages.

Similarly, OLPC's "public" dev.laptop.org server allows trusted contributors to place RPMs in their home directory, which then get aggregated at (e.g.) http://xs-dev.laptop.org/~dsd/repos/f11/ These packages are not vetted or monitored, so when development reaches a late stage, there is obvious reason to switch to a frozen repos system where approved and reviewed packages are added manually. Therefore separate frozen repos are created for these packages, and the build system switches from xs-dev to mock as its source of OLPC-local packages.

During the last few weeks of development, new RPMs are added to these repositories by hand via other undocumented processes, but the rate of change is typically low.

mock.laptop.org retains these frozen RPM trees forever, providing a means for the release to be exactly rebuilt from the same set of packages.

Existing setup and problems

mock.laptop.org has a single git tree of various distinct branches, including entire copies of Fedora, fedora updates, and OLPC-added RPMs.

Many of these branches have very little common history, because every time the Fedora version changes, close to 100% of RPMs are changed.

This git tree is really huge and is quite difficult to work with (git is not good with binary files). Given the lack of common history between branches, it seems like an odd model too.

The git tree is made available over gitweb, and some clever lighthttp configuration makes regular http:// requests be passed through gitweb's "blob" view.

We are now out of disk space on the existing system. The git repo is more than 100GB in size.

Proposed system

Core Fedora repositories

There is no reason to have the core Fedora repos (the ones made at point of Fedora release) under version control, because they are fixed by definition. They don't change.

So, Fedora repositories will be stored on disk outside of revision control, in the following structure:

• RPMS/ (copy of all the RPMs)
• SRPMS/ (copy of all the SRPMs)
• repodata/ (generated by createrepo)
• comps.xml (copied from Fedora)

Delta RPMs and debuginfo packages are not included.

There are sometimes a handful of packages that don't change between Fedora releases, so these repositories should be constructed using hard links based on the previous Fedora release present on this server, before using rsync to download the new Fedora release.

So the process for Fedora 14 will be:

# cp -a -l koji.dist-f11 koji.dist-f14
# cd koji.dist-f14
# rsync -av --progress --delete rsync://mirrors.rit.edu/fedora-enchilada/linux/releases/14/Everything/i386/os/Packages/ RPMS/
# rsync -av --progress --delete --exclude repodata rsync://mirrors.rit.edu/fedora-enchilada/linux/releases/14/Everything/source/SRPMS/ SRPMS/
# rm -f comps.xml
(now download new comps.xml manually from http://mirrors.rit.edu/fedora/linux/releases/14/Everything/i386/os/repodata/
# createrepo -g comps.xml -d .

Fedora-updates repositories

Fedora updates are also included in OLPC OS releases, generally taken as a snapshot at a particular point in time (quite late in the release process). OLPC OS development is based on updates-testing, therefore when a snapshot of fedora-updates is taken, a few packages must be brought in manually from updates-testing in order to ensure a consistent build with the development builds that came before it.

As these trees are largely static, they will not be kept in revision control. The process to create such a snapshot is once again based on a hardlinked tree of RPMs from the previous fedora-updates repository, for purposes of disk space saving.

The structure is as follows:

• RPMS/ (copy of all the RPMs)
• SRPMS/ (copy of all the SRPMs)
• repodata/ (generated by createrepo)
• comps.xml (copied from Fedora)
• ChangeLog (hand written by OLPC developer)

Delta RPMs and debuginfo packages are not included.

For example, for 11.2.0's snapshot of fedora-updates:

# cp -a -l koji.dist-f11-updates-10.1.2 koji.dist-f14-updates-11.2.0
# cd koji.dist-f14-updates-11.2.0
# rsync -av --progress --delete --exclude debug --exclude repodata --exclude drpms rsync://mirrors.rit.edu/fedora-enchilada/linux/updates/14/i386/ RPMS/
# rsync -av --progress --delete --exclude repodata rsync://mirrors.rit.edu/fedora-enchilada/linux/updates/14/SRPMS/ SRPMS/
# rm -f comps.xml
(now download new comps.xml manually from http://mirrors.rit.edu/fedora/linux/updates/14/i386/repodata/ )
# createrepo -g comps.xml -d .
# rm -f ChangeLog

Now write an initial ChangeLog entry, including your name, the time and date, and the fact that you pulled in updates.

Now we must account for any packages from updates-testing which are being shipped in the latest development release before the switch to frozen repos. The easiest way to do this is to produce a local OS build based on the frozen repos, and then do a diff of the packages.txt output against the one from the latest development build. Look for differences where packages are being downgraded in your local build. You then know which packages need syncing in from updates-testing.

This is a manual process but usually not many packages fall into this category (I think it was about 10 last time). After examining the diff to see which packages need pulling, the process for each package is:

(from inside the koji.dist-f14-updates-11.2.0 directory created above)

1. If there is an existing (but older) update for the package, remove it from RPMS and SRPMS
2. Download the desired version from http://koji.fedoraproject.org - put the .rpm in RPMS and the SRPM in .srpms (for packages with subpackages, all subpackages must be synced like this)

Finally:

# createrepo -g comps.xml -d .

Now edit the ChangeLog, add a new entry with your name, the time and date, and a list of packages that you manually pulled in.

As development continues, new Fedora updates will occasionally want to be brought in in response to specific bugs. The above process (of syncing in packages from updates-testing) can be followed. The ChangeLog entry that you add should include a reference to the ticket that is being fixed.

OLPC-local repositories

Unlike Fedora, the packages that OLPC adds are not built by a centralized build system which archives old versions. And unlike the Fedora repository clones, it is quite likely that there will be a fair degree of change in these repositories during the final weeks of development. So, revision control is called for.

As before, there will be one repository for "generic" OLPC packages, and then one repository per targetted laptop model for the laptop-specific packages. However, unlike before, each of these repositories will be separate git trees (after all, they do not share history). These repositories will be created on the server by the release manager.

The layout of each repository will be:

• RPMS/
• SRPMS/
• metadata/ (created by createrepo)

These repositories will be available over git:// and ssh://. Contributors will have push access to the relevant repositories (but this task will usually be handled exclusively by the release manager).