Build system

Output from the Build System

The build system produces things:

1. An RPM repository, currently located under the repository root

2. The output of the build system is currently located under the streams root

Maintainers

1. Scott Ananian (c_scott)

2. Dennis Gilmore (dgilmore)

3. Bernardo Innocenti (_bernie)

4. Michael Stone (m_stone)

Build Branches

Frequently, we find ourselves needing to integrate complicated changes that cut across packages. To make this easier, we make it very easy to 'branch' a build in which the integration can be performed safely, without interfering with other work. If you are trying to perform a delicate piece of integration for which it would be helpful to have a separate build, please contact one of the build system maintainers.

We currently have five build branches arranged in an inheritance tree like so:

 kernel <- xtest <- datastore <- sugar <- rainbow <- joyride.

Instructions for Use

Before getting started, you will need an account on dev.laptop.org (See Project hosting).

To add a completely new piece of software (either a new activity or a new package) to a build, three steps are necessary.

1. Include a source code package (.xo or .src.rpm) and a binary package (.xo or .rpm) in your dropbox (which is located at ~/public_rpms/<build>/ on dev.laptop.org). You will need to create the folder, here is an example for joyride:

mkdir -p ~/public_rpms/joyride

Note: do not remove the .xpm's or .xo's from the public_rpms location, not even after they already showed up in an image: public_rpms is not a temporary place where the build systems grab them; it will grab them from that location every run.

2. Update the change log to describe the changes made by the packages you are introducing. We accumulate changelog entries for ~/public_rpms/<build>/ChangeLog and ~/public_rpms/<build>/*.changes, so you can use a separate .changes file for each major group of packages you maintain.

3. Where possible, it's best to add your package as a new dependency of an existing package, so that the reason for the inclusion of this package is clear. If this is not possible, file a bug in trac with component 'distro' and owner 'cscott' requesting that the new package or activity be added to the build.

To verify that a package (here Web-72.xo) has been picked up you can do the following:

First you can look in the repos/joyride if there was a Web-*.xo present. Another interesting file in the same directory is the ChangeLog which is the aggregate ChangeLog that describes all of the packages pulled in from every dropbox. It contains warnings emitted when some problem prevents your RPMs from being picked up. Finally, you can read the buildlog to check if Web-72.xo was installed.

Bert made a script that aggregates changelogs as joyride-pkgs.html, which is useful for comparing build versions.

ChangeLog Format

The format of this ChangeLog is

 <pkg-lines>      (containing ".xo", ".rpm")                                                                                                                                  
 <detail-lines>   (not containing the string ".xo", ".rpm", "--")                                                                                                             
 -- <signature>   (-- is the line marker)                                                                                                                                     
                                                                                                                                                                              
 <pkg>                                                                                                                                                                        
 <pkg>.rpm                                                                                                                                                                    
 <detail>                                                                                                                                                                     
 -- <signature>

An example:

 Etoys-61.xo
 etoys-2.2.1710-1.noarch.rpm
 etoys-2.2.1710-1.src.rpm
 
   * changed activity launch to match new Sugar
   * words in tiles separated by spaces not camelCase
   * adds "distance to" and "bearing to" tiles
   * better .pot sort order
 
   -- Bert Freudenberg <bert@freudenbergs.de>  Tue, 16 Oct 2007
 
 squeak-vm-3.9-12olpc1.i386.rpm
 squeak-vm-3.9-12olpc1.src.rpm
 
   * big merge with trunk
   * unicode key events
   * fixed dbus plugin
 
   -- Bert Freudenberg <bert@freudenbergs.de>  Mon, 15 Oct 2007

Common Mistakes

You must use different time stamps for each ChangeLog entry on the -- author line.

If there are multiple authors of a single build-patch, you should list each person's contribution like so:

 pkg.rpm
   * change
   -- author 1
 pkg.rpm
   * change
   -- author 2

rather than like

 pkg.rpm
   * change
   -- author 1
   * change
   -- author 2

until we improve the ChangeLog parser.

Implementation Details

The build system we want to have consists of the following components:

                              /------\
                             | F7 yum |--------+
                             | repos  |        |
                              \------/         V
 /--------\    +--------+   +----------+   +---------+    /------\
| upstream |   | distro |   | pkg build|   |compose  |   | stable |
|    src   |-->|  SCM   |-->|  system  |-->| tools   |-->| rainbow|
|   repos  |   | (CVS)  |   |   (Koji) |   |(joyride)|   | xtest  |
 \--------/    +--------+   +----------+   +---------+    \------/
                ^  ^                           ^
 /--------\     |  |                           |
 |patches |-----+  |      + - - - - - - - +    |
 \--------/        |      |  dropbox      |    |
                   +----->|  system       |----+
 /--------\        |      | (public_rpms) |
|   olpc   |       |      + - - - - - - - +
|  sources |-------+         
|   (GIT)  |
 \--------/


|-----------| |----------------------------| |-----------------------|
   sources         package maintenance          package composition
   CVS,GIT         Koji, Mock, rpmbuild       mash, pilgrim, ChangeLogs

The one we actually have is fairly similar but the package-construction step is done partially in Koji and partially by individual package maintainers using tools like Mock and rpmbuild.

Moving Forward

• Build system requirements

• Build system meeting

• Bernie's build system workflow proposal

Source Code: Patches Welcome

Dropbox Mechanism

Source code

The dropbox mechanism is a way for people who don't have the resources to work through koji to supply us with activities and RPMs.

It is implemented in two pieces, one running on the dropbox-host and one running on the build-host. The dropbox-host simply provides an rsync daemon that will transfer packages to the builder on request.

The build-host uses the program 'collect-pkgs.py' to aggregate packages and change-entries into an RPM repository for composition by pilgrim.

Pilgrim

For information on current-day OLPC software build systems, see Build system.

WARNING: The content of this section is considered DEPRECATED and OBSOLETE It is preserved for historical or documenting reasons. Pilgrim is a disk-image compiler. This means that it converts collections of packages (RPMs) and hacks into clean-installable disk images for the XO. It is typically run on a dedicated Linux server or virtual machine like xs-dev.laptop.org or pilgrim.laptop.org. 2010 OLPC builds use the OS Builder utility instead. Output from the Build System The build system produces things: An RPM repository, currently located under the repository root The output of the build system is currently located under the streams root at http://pilgrim.laptop.org/~pilgrim/olpc/streams/ for update.1, and somewhere else (maybe http://xs-dev.laptop.org/~cscott/olpc/streams/ for joyride. Files produced For an image called <imagename>, the files you will find inside are... fs.zip - signatures file fs.zip.md5 - the md5 checksum of fs.zip <imagename>.crc - the crc of the image <imagename>.crc.md5 - the md5 checksum of the crc of the image <imagename>.img - the image file itself <imagename>.img.md5 - the md5 checksum of the image file If the image contains Activities and other customizations on top of a build, you will also find the following files. Note that this is rare and typically only done once a year for images shipped on G1G1 XOs. <imagename>.zip - the customization key included in the image. <imagename>.zip.md5 - the md5 checksum of the customization key included in the image. Implementation Details The build system we want to have consists of the following components: /------\ | F7 yum |--------+ | repos | | \------/ V /--------\ +--------+ +----------+ +---------+ /------\ | upstream | | distro | | pkg build| |compose | | stable | | src |-->| SCM |-->| system |-->| tools |-->| rainbow| | repos | | (CVS) | | (Koji) | |(joyride)| | xtest | \--------/ +--------+ +----------+ +---------+ \------/ ^ ^ ^ /--------\ | | | |patches |-----+ | + - - - - - - - + | \--------/ | | dropbox | | +----->| system |----+ /--------\ | | (public_rpms) | | olpc | | + - - - - - - - + | sources |-------+ | (GIT) | \--------/ |-----------| |----------------------------| |-----------------------| sources package maintenance package composition CVS,GIT Koji, Mock, rpmbuild mash, pilgrim, ChangeLogs The one we actually have is fairly similar but the package-construction step is done partially in Koji and partially by individual package maintainers using tools like Mock and rpmbuild. Details Source code Pilgrim is composed of two large and two small scripts and many configuration files. pilgrim # user-interface pilgrim-autobuild # implementation build-one # build the current branch make-repos # touch up yum repositories The most important branch is "autobuild". This is where changes deemed suitable for all build branches should be merged. See Pilgrim Commit Policy. The other branches - joyride, meshtest, xtest, and rainbow store branch-specific data like which yum repositories to compose into the image, which packages to pull, which activities to include, and so on. These data are mostly recorded in streams.d/olpc-branch.conf # a few important config variables streams.d/olpc-development-yum-install.conf # yum repositories to compose, package exclusions streams.d/olpc-development.stream # lists of packages and activities to install Questions: How do you enumerate the available branches? I.e. if you don't want to rely on the documentation being kept up to date... Where can one find the streams.d directory? Motivation This tutorial explains how you can build a completely customized OS image which can be used with: Emulators such as QEMU and VMware Laptops running in developer mode (see Activation and developer keys about running your xo in developer mode and Clean-install procedure about installing OS images) Prerequisites The following procedure has been tested on a Fedora 8 x86_64, but should work just fine on Fedora 7 and on i386. Any Linux distribution capable of running yum should be able to run Pilgrim, maybe with minor adjustments. The intended audience of this document is an OS developer with some experience in rebuilding RPMs and managing yum repositories. Preparation Before installing pilgrim, you need to install mtd-utils. sudo yum install mtd-utils git clone git://dev.laptop.org/users/cscott/pilgrim cd pilgrim make sudo make install Note: this will litter your system with pilgrim files, but you can clean it up later with make uninstall. Prepare your custom repositories Build your custom RPMs and place them in a directory accessible via http. It is recommended, but not necessary, to adopt the standard Fedora layout: i386/os/ - i386 and noarch packages i386/debug/ - debuginfo packages source/ - src rpms Create or update the RPM metadata information for your repositories: createrepo i386/os createrepo i386/debug createrepo source Pilgrim customization Adding your yum repositories Edit streams.d/olpc-development-yum-install.conf and append your yum repository (you only need the one containing the binary RPMs here). Mine looks like this: [olpc-bernie] name=OLPC bernie - i386 baseurl=http://www.codewiz.org/pub/olpc-bernie/i386/os/ enabled=1 gpgcheck=0 Note that I've explicitly specified i386 because the $basearch variable would expand to x86_64 on my host. If you provide the full set of OS packages in your repository, then you should remove the default repositories altogether. Adding repositories for the target Edit streams.d/olpc-development.stream, and find the line saying "Writing out configuration for OLPC yum repo". The repositories listed here will appear in /etc/yum.repos.d on the target and are only used when the image is booted on the target. Add your yum repositories below the official ones. This is what I do for xtest: # keep this in sync with same repo defined in olpc-development-yum-install.conf cat <<EOF > $INSTALL_ROOT/etc/yum.repos.d/olpc-development.repo [olpc_development] name=OLPC development baseurl=http://koji.fedoraproject.org/static-repos/dist-olpc2-build-current/i386/ enabled=1 gpgcheck=0 EOF cat <<EOF > $INSTALL_ROOT/etc/yum.repos.d/olpc-bernie.repo [olpc-bernie] name=OLPC bernie - \$basearch baseurl=http://www.codewiz.org/pub/olpc-bernie/\$basearch/os/ enabled=1 gpgcheck=0 [olpc-bernie-debuginfo] name=OLPC bernie - \$basearch - Debug baseurl=http://www.codewiz.org/pub/olpc-bernie/\$basearch/debug/ enabled=0 gpgcheck=0 [olpc-bernie-source] name=OLPC bernie - Source baseurl=http://www.codewiz.org/pub/olpc-bernie/source/ enabled=0 gpgcheck=0 EOF Adding and removing packages The list of packages that will be installed is in streams.d/olpc-development.stream. If your spec files are correct, you will only need to list the leaf packages here and yum will pull in all their dependencies automatically. Further reading Read README.olpc in the root directory of Pilgrim for further information. The short story is: (become root) ./pilgrim-autobuild --config-dir . --stream olpc-development --dest-dir . If you're only interest in the jffs2 image, do: ./pilgrim-autobuild --config-dir . --stream olpc-development --dest-dir . --variant devel_jffs2 To install OS images see Clean-install procedure. Puritan For information on current-day OLPC software build systems, see Build system. The currency of this article or section may be limited by out-of-date information. There may be relevant discussion on its talk page Puritan Source code. README. Latest version: ui: e7a6bdb6, compilation: 8b944b2d. Puritan is a disk-image compiler which converts source material including packages, activities, and hacks into installable disk images. It consists of two pieces: a UI and a family of compilations (example). The compilations are simple Python programs which populate a filesystem with the materials you supply, then format it for distribution. The UI runs the compilations in a controlled environment created by Mock and configured according to the compilation's bootstrap and dependencies files. Puritan was developed as a replacement for Pilgrim but was never officially adopted by OLPC. For details on the active OLPC build system, see Build system. Principles Puritan is based on several fundamental principles. Tools should be usable for both decentralized and centralized patterns of development. People **will** want to make unpackaged changes to their builds. Caching matters. Interactive development matters. Build reproducibility matters. Error-detection, handling, and cleanup matter. People working on the OLPC build system will become accustomed to git and python. Instructions Preparation To use Puritan, you need to install dependencies: # NB: mock >= 0.9.7 is needed. # debian: sudo apt-get install mock git-core # fedora: sudo yum install mock git-core # both sudo usermod -a -G mock $USER newgrp mock and to check out and run a 'ui' and a 'compilation'. (git clone git://dev.laptop.org/users/mstone/puritan compilation; cd compilation; git branch --track 767 origin/767; git checkout 767) git clone git://dev.laptop.org/users/mstone/puritan ui python2.5 ui/puritan/main.py # read help sudo python2.5 ui/puritan/main.py -v build ./compilation HEAD ./results -- -i Interactive Debugging The '-i' option above enables an interactive error-handling mode. If an error occurs, with the most recent compilation and ui, you will be dropped directly into bash inside the buildroot. to resume the compilation, type exit 0 to quit the compilation, type exit 1 with earlier compilations which still support interactive error-handling: instructions Other Notes Fedora Note: if you encounter permission errors when running this command, you may need to: sudo setenforce 0 # disable selinux Note: a word of warning: while _my_ uis and compilations will treat your computer nicely, other people's might not. Caveat emptor. Ubuntu mock_0.9.7-2 on ubuntu requires [1] in order to run. Additionally, if you run puritan as root, you should either usermod -a -G mock -u $SUDO_UID unset SUDO_UID # or All Some puritan compilations now feature a 'download' operation which can be used to download all packages that the compilation will install for later (offline) use: sudo python2.5 ui/puritan/main.py -v download ./compilation HEAD ./results This command will generate a program in the results directory which can be used to download the packages. Configuration Puritan is highly and easily configurable by means of small modifications to the files and directories contained in each compilation. Detailed suggestions on how to configure Puritan are available. Notes and Tricks Use the source, Luke! (It was written to be read!) with some uis and compilations, various subcommands (e.g. 'build') may fail because /etc/mock/fedora-9-i386.cfg is not present in Debian's mock package 0.9.7-2 one workaround is to create it from http://teach.laptop.org/~mstone/fedora-9-i386.cfg. Mock and SElinux do not interact very well. at least one workaround is discussed on the Fedora MockTricks] page. the author simply disables selinux with sudo setenforce 0 in order to avoid the hassle. Meld is a great tool for diffing the filesystem trees produced by the 'produce_tree' build stage and for comparing your builds to those made by others. Git will be your friend, but, to get the most out of the relationship, you're going to have to invest in it. Ask for help when you need it! gitk --all or git log will show you lots of other compilations you could run! The latest compilation now automatically caches packages in a directory called 'smart/packages' inside the compilation. Some other state is also cached, so if things start behaving strangely, try deleting the 'smart' directory. Help Out Puritan was designed from the ground up to support decentralized development because its author felt that deployment autonomy was going to be critical to the success of the OLPC project. Therefore, please use your autonomy! In particular, please help improve puritan by: Using it! As much as you can! Produce cool disk images! Publish your patches! (patch review available on request). Making puritan work on your platform - it's only dependencies are python2.5, git-core, and mock! Maintaining the devel_ext3 compilation Adding some reasonable automatic package or buildroot caching system so that it runs faster without impairing build repeatability Improving the UI with commands for manipulating compilations, or for diffing builds, or for profiling compilations, or ... School Server Build Tools See XS Building Software for details.