RPM Dropbox

Introduction

OLPC's dropbox system allows contributors to submit RPM packages for inclusion in XO software releases that are in development stages. The contributor places the RPM files in a specific place under his home directory on OLPC's development server, and the dropbox system then aggregates them alongside packages from other users, turns them into a yum repository, and publishes them at http://xs-dev.laptop.org/~dsd/repos/

During initial development of OLPC OS releases, the build system uses a selection of these repositories a source of packages. The exact set of dropboxes used for each development stream is noted on the release page for the in-development release in question. However, just placing a new package in the dropbox is not enough for inclusion of an entirely new package; the build system lists the packages to be included in the builds, this must be updated so that it gets pulled in.

These repositories are aggregated and published every 30 minutes.

Each repository includes a "filelist.txt" file which shows the sources (i.e. which user) of each package in the dropbox.

If a package gets placed in the dropbox with the same name as a package in Fedora, the version in the dropbox takes precedence -- even if the Fedora repositories include a newer version of the same package. This provides OLPC with a simple method to make modifications to Fedora packages.

Packages in the dropbox are not vetted before being automatically included in development builds. However, once the software release reaches stabilization stages, OLPC switches to using a "frozen repository" system where each and every change is vetted and pushed by the release manager. Dropboxes are no longer used by the build system at that point, but tend to stick around as a way of pushing your new customized packages to the release manager for review and inclusion in the builds. See the Release Process documentation for the details.

Please don't use this system!

This system should only be used in exceptional cases!

OLPC does not have a large software development team and anything put in these repositories becomes entirely our problem.

If you want to add a new package to the system, you should get it packaged directly in Fedora, and get it included in OLPC builds via the regular Fedora package channels.

If you need to fix a bug in a package that comes from Fedora, you may be tempted to whip up a patch, add it to the RPM, and put it in the RPM dropbox. However, the correct approach is to get the bug fixed properly upstream and in Fedora.

For the build system, packages in the dropbox take precedence over Fedora repositories (even if Fedora has a newer version). That means that the following situation is possible:

1. You decide take foobar-1.0-1 from Fedora, fork it as foobar-1.0-1.olpc to add a patch that we need to solve a bug
2. foobar-1.1 comes out, fixing a really important security or data loss bug
3. Fedora pushes foobar-1.1 immediately due to the severity of the bug
4. OLPC does not receive this new version due to the presence of foobar-1.0-1.olpc in the dropbox, and nobody on OLPC's small team notices the severe bug

OLPC does not only work on a short-term basis. Every time we change Fedora version, we must rebuild and rebase everything in the dropbox. This is often a time consuming process. The net effect is that if you put a package in the dropbox, you take away development time from the following development cycle, as you must spend time bringing forward all the non-upstream changes.

As a rule of thumb, when you put something in the dropbox, think about who is going to maintain it in the next 6 months, 1 year, 5 years (hint: this becomes YOUR responsibility). We often only use the dropbox on a short-term basis with a solid route of getting the real fix into Fedora, because forking packages is too much of a drain on our resources.

Usage

Accounts on dev.laptop.org are available to proven contributors, see the Project hosting page.

Dropbox usage is simply maintaining a collection of packages under a public_rpms directory in your home directory. Inside public_rpms, create a directory with the name of the dropbox you want to contribute to, and put RPM files inside.

Please submit source RPM (SRPM) too

When adding a package to the dropbox, please submit the corresponding .src.rpm file to the same dropbox. This allows people to identify the changes you have made (if any), do future rebuilds, etc.

Adding a package

If this is the first time you are contributing to the dropbox in question, you must create the parent directory. For example, for the 'f14' dropbox:

ssh user@dev.laptop.org
mkdir -p public_rpms/f14

Now it is just a case of putting files in that directory. You could do this from your local machine with the scp command, e.g.

scp foobar-1.1-1.fc14.i586.rpm foobar-1.1-1.fc14.src.rpm dev.laptop.org:public_rpms/f14

Removing a package

Removing a package from the dropbox is as simple as removing it from your own public_rpms folder. Simply login to dev.laptop.org and use the rm command to remove the files. Note that dropbox files are not permanently archived, so removing files usually equates to deleting them forever; take a backup if you think you might need this file again.

How it works

This section is only for curious people or people performing maintenance/development on the system. You don't need to understand this if you just want to use the dropbox system.

On dev.laptop.org, a script at /etc/find-rsync-rpms creates an rsyncd config file, exposing one rsync module per user account that has a public_rpms directory, e.g.

{{{ [dsd]

path = /home/dsd/public_rpms
comment = public rpms exported by dsd
read only = true
timeout = 600

}}}

(we aren't exactly sure how or if this script is run automatically)

On xs-dev, the do-repocollect script from http://dev.laptop.org/git/users/dsd/rpm-dropbox is run every 30 minutes via cron. This script examines all available rsync modules published on dev.laptop.org, pulls in all the RPMs, creates yum repositories, and outputs them in a directory served by apache. The switch from old to new repositories is done atomically.