Build system
From OLPC
Contents |
[edit] Output from the Build System
The build system produces things:
1. An RPM repository, currently located under the repository root
2. The output of the build system is currently located under the streams root at http://pilgrim.laptop.org/~pilgrim/olpc/streams/ for update.1, and somewhere else (maybe http://xs-dev.laptop.org/~cscott/olpc/streams/ for joyride.
[edit] Maintainers
1. Scott Ananian (c_scott)
2. Dennis Gilmore (dgilmore)
3. Bernardo Innocenti (_bernie)
4. Michael Stone (m_stone)
[edit] Build Branches
Frequently, we find ourselves needing to integrate complicated changes that cut across packages. To make this easier, we make it very easy to 'branch' a build in which the integration can be performed safely, without interfering with other work. If you are trying to perform a delicate piece of integration for which it would be helpful to have a separate build, please contact one of the build system maintainers.
We currently have five build branches arranged in an inheritance tree like so:
kernel <- xtest <- datastore <- sugar <- rainbow <- joyride.
[edit] Instructions for Use
Before getting started, you will need an account on dev.laptop.org (See Project hosting).
To add a completely new piece of software (either a new activity or a new package) to a build, three steps are necessary.
1. Include a source code package (.xo or .src.rpm) and a binary package (.xo or .rpm) in your dropbox (which is located at ~/public_rpms/<build>/ on dev.laptop.org). You will need to create the folder, here is an example for joyride:
mkdir -p ~/public_rpms/joyride
Note: do not remove the .xpm's or .xo's from the public_rpms location, not even after they already showed up in an image: public_rpms is not a temporary place where the build systems grab them; it will grab them from that location every run.
2. Update the change log to describe the changes made by the packages you are introducing. We accumulate changelog entries for ~/public_rpms/<build>/ChangeLog and ~/public_rpms/<build>/*.changes, so you can use a separate .changes file for each major group of packages you maintain.
3. Where possible, it's best to add your package as a new dependency of an existing package, so that the reason for the inclusion of this package is clear. If this is not possible, file a bug in trac with component 'distro' and owner 'cscott' requesting that the new package or activity be added to the build.
To verify that a package (here Web-72.xo) has been picked up you can do the following:
First you can look in the repos/joyride if there was a Web-*.xo present. Another interesting file in the same directory is the ChangeLog which is the aggregate ChangeLog that describes all of the packages pulled in from every dropbox. It contains warnings emitted when some problem prevents your RPMs from being picked up. Finally, you can read the buildlog to check if Web-72.xo was installed.
- Bert made a script that aggregates changelogs as joyride-pkgs.html, which is useful for comparing build versions.
[edit] ChangeLog Format
The format of this ChangeLog is
<pkg-lines> (containing ".xo", ".rpm")
<detail-lines> (not containing the string ".xo", ".rpm", "--")
-- <signature> (-- is the line marker)
<pkg>
<pkg>.rpm
<detail>
-- <signature>
An example:
Etoys-61.xo etoys-2.2.1710-1.noarch.rpm etoys-2.2.1710-1.src.rpm * changed activity launch to match new Sugar * words in tiles separated by spaces not camelCase * adds "distance to" and "bearing to" tiles * better .pot sort order -- Bert Freudenberg <bert@freudenbergs.de> Tue, 16 Oct 2007 squeak-vm-3.9-12olpc1.i386.rpm squeak-vm-3.9-12olpc1.src.rpm * big merge with trunk * unicode key events * fixed dbus plugin -- Bert Freudenberg <bert@freudenbergs.de> Mon, 15 Oct 2007
[edit] Common Mistakes
You must use different time stamps for each ChangeLog entry on the -- author line.
If there are multiple authors of a single build-patch, you should list each person's contribution like so:
pkg.rpm * change -- author 1 pkg.rpm * change -- author 2
rather than like
pkg.rpm * change -- author 1 * change -- author 2
until we improve the ChangeLog parser.
[edit] Implementation Details
The build system we want to have consists of the following components:
/------\
| F7 yum |--------+
| repos | |
\------/ V
/--------\ +--------+ +----------+ +---------+ /------\
| upstream | | distro | | pkg build| |compose | | stable |
| src |-->| SCM |-->| system |-->| tools |-->| rainbow|
| repos | | (CVS) | | (Koji) | |(joyride)| | xtest |
\--------/ +--------+ +----------+ +---------+ \------/
^ ^ ^
/--------\ | | |
|patches |-----+ | + - - - - - - - + |
\--------/ | | dropbox | |
+----->| system |----+
/--------\ | | (public_rpms) |
| olpc | | + - - - - - - - +
| sources |-------+
| (GIT) |
\--------/
|-----------| |----------------------------| |-----------------------|
sources package maintenance package composition
CVS,GIT Koji, Mock, rpmbuild mash, pilgrim, ChangeLogs
The one we actually have is fairly similar but the package-construction step is done partially in Koji and partially by individual package maintainers using tools like Mock and rpmbuild.
[edit] Moving Forward
[edit] Source Code: Patches Welcome
[edit] Dropbox Mechanism
The dropbox mechanism is a way for people who don't have the resources to work through koji to supply us with activities and RPMs.
It is implemented in two pieces, one running on the dropbox-host and one running on the build-host. The dropbox-host simply provides an rsync daemon that will transfer packages to the builder on request.
The build-host uses the program 'collect-pkgs.py' to aggregate packages and change-entries into an RPM repository for composition by pilgrim.
[edit] Pilgrim
Please add text to answer the following questions:
- what is Pilgrim?
- is it already on XO laptops or do you need to install it?
- What problem does it solve?
- Who is expected to use it?
- What are the typical use cases for Pilgrim?
[edit] Details
Pilgrim is composed of two large and two small scripts and many configuration files.
pilgrim # user-interface pilgrim-autobuild # implementation build-one # build the current branch make-repos # touch up yum repositories
The most important branch is "autobuild". This is where changes deemed suitable for all build branches should be merged. See Pilgrim Commit Policy.
The other branches - joyride, meshtest, xtest, and rainbow store branch-specific data like which yum repositories to compose into the image, which packages to pull, which activities to include, and so on.
These data are mostly recorded in
streams.d/olpc-branch.conf # a few important config variables streams.d/olpc-development-yum-install.conf # yum repositories to compose, package exclusions streams.d/olpc-development.stream # lists of packages and activities to install
Questions:
- How do you enumerate the available branches? I.e. if you don't want to rely on the documentation being kept up to date...
- Where can one find the streams.d directory?
[edit] Puritan
Source code. README. Latest version: 0.4.
Puritan is a disk-image compiler which converts source material including packages, activities, and hacks into installable disk images. It consists of two pieces: a UI and a family of compilations (example). The compilations are simple Python programs which populate a filesystem with the materials you supply, then wrap it up with a bow-tie. The UI runs the compilations in a controlled environment created by Mock and configured according to the compilation's bootstrap and dependencies files.
[edit] Fedora Instructions
The Puritan UI is invoked by installing a puritan rpm (built from the ui branch)
wget -O puritan.rpm http://dev.laptop.org/~mstone/releases/RPMS/puritan-0.4-1.olpc2.noarch.rpm sudo rpm -Uvh puritan.rpm
then by checking out a 'puritan compilation' to be run by /usr/bin/puritan
$(git clone git://dev.laptop.org/users/mstone/puritan compilation; cd compilation; git checkout origin/devel_jffs2) puritan -v ./compilation HEAD ./results build
If you want to download a set of RPMs for post-processing (for example, because you want to layer a risky feature on top of them), you can also run
puritan -v ./compilation HEAD ./results download
Your results should appear in ./results.
[edit] Debian Instructions
Note: yum seems to be broken on Debian at the moment, which is preventing this recipe from working. :( --Michael Stone 03:02, 7 March 2008 (EST)
sudo apt-get install mock git-core sudo usermod -a -G mock $USERNAME git clone git://dev.laptop.org/users/mstone/puritan ui $(git clone git://dev.laptop.org/users/mstone/puritan compilation; cd compilation; git checkout origin/devel_jffs2) /usr/bin/python2.5 ui/puritan/main.py ./compilation HEAD ./results
[edit] Help Out
Finally, please help improve puritan by:
- Making puritan work on your platform - it's only dependencies are python2.5, git-core, and mock!
- Maintaining the devel_ext3 compilation
- Adding some reasonable package or buildroot caching system so that it runs faster without impairing build repeatability
- Improving the UI with commands for manipulating compilations, or for diffing builds, or for profiling compilations, or ...
[edit] School Server Build Tools
See XS Building Software for details.
