Pilgrim: Difference between revisions

From OLPC
Jump to navigation Jump to search
m (mention OS Builder)
No edit summary
Line 1: Line 1:
<b><font color=red><big>For information on current-day OLPC software build systems, see [[Build system]].</big></font></b>

{{Deprecated}}

Pilgrim is a disk-image compiler. This means that it converts collections of packages (RPMs) and hacks into [[Clean-install|clean-installable]] disk images for the XO. It is typically run on a dedicated Linux server or virtual machine like xs-dev.laptop.org or pilgrim.laptop.org. 2010 OLPC builds use the [[OS Builder]] utility instead.
Pilgrim is a disk-image compiler. This means that it converts collections of packages (RPMs) and hacks into [[Clean-install|clean-installable]] disk images for the XO. It is typically run on a dedicated Linux server or virtual machine like xs-dev.laptop.org or pilgrim.laptop.org. 2010 OLPC builds use the [[OS Builder]] utility instead.


== Output from the Build System ==
: ''When the [[OLPC:Volunteer Infrastructure Group]] is up and running, we should link to pages describing the build infrastructure in detail.''

The build system produces things:

# An RPM repository, currently located under the [http://xs-dev.laptop.org/~cscott/repos repository root]
# The output of the build system is currently located under the streams root at [http://pilgrim.laptop.org/~pilgrim/olpc/streams/ http://pilgrim.laptop.org/~pilgrim/olpc/streams/] for update.1, and somewhere else (maybe [http://xs-dev.laptop.org/~cscott/olpc/streams/ http://xs-dev.laptop.org/~cscott/olpc/streams/] for joyride.

=== Files produced ===

For an image called <code><imagename></code>, the files you will find inside are...

# '''fs.zip''' - signatures file
# '''fs.zip.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of fs.zip
# '''<imagename>.crc''' - the [http://en.wikipedia.org/wiki/Cyclic_redundancy_check crc] of the image
# '''<imagename>.crc.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the [http://en.wikipedia.org/wiki/Cyclic_redundancy_check crc] of the image
# '''<imagename>.img''' - the image file itself
# '''<imagename>.img.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the image file

If the image contains Activities and other customizations on top of a build, you will also find the following files. Note that this is rare and typically only done once a year for images shipped on G1G1 XOs.

# '''<imagename>.zip''' - the [[customization key]] included in the image.
# '''<imagename>.zip.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the [[customization key]] included in the image.


== Implementation Details ==

The build system we want to have consists of the following components:

<pre>
/------\
| F7 yum |--------+
| repos | |
\------/ V
/--------\ +--------+ +----------+ +---------+ /------\
| upstream | | distro | | pkg build| |compose | | stable |
| src |-->| SCM |-->| system |-->| tools |-->| rainbow|
| repos | | (CVS) | | (Koji) | |(joyride)| | xtest |
\--------/ +--------+ +----------+ +---------+ \------/
^ ^ ^
/--------\ | | |
|patches |-----+ | + - - - - - - - + |
\--------/ | | dropbox | |
+----->| system |----+
/--------\ | | (public_rpms) |
| olpc | | + - - - - - - - +
| sources |-------+
| (GIT) |
\--------/


|-----------| |----------------------------| |-----------------------|
sources package maintenance package composition
CVS,GIT Koji, Mock, rpmbuild mash, pilgrim, ChangeLogs
</pre>

The one we actually have is fairly similar but the package-construction step is done partially in [https://fedoraproject.org/wiki/Koji Koji] and partially by individual package maintainers using tools like [https://fedoraproject.org/wiki/Projects/Mock Mock] and rpmbuild.


More information on pilgrim is at [[Building custom images]].


==== Details ====
==== Details ====

Revision as of 16:47, 8 February 2011

For information on current-day OLPC software build systems, see Build system.

Stop hand.png WARNING:
The content of this section is considered
DEPRECATED and OBSOLETE
It is preserved for historical or documenting reasons.

Pilgrim is a disk-image compiler. This means that it converts collections of packages (RPMs) and hacks into clean-installable disk images for the XO. It is typically run on a dedicated Linux server or virtual machine like xs-dev.laptop.org or pilgrim.laptop.org. 2010 OLPC builds use the OS Builder utility instead.

Output from the Build System

The build system produces things:

  1. An RPM repository, currently located under the repository root
  2. The output of the build system is currently located under the streams root at http://pilgrim.laptop.org/~pilgrim/olpc/streams/ for update.1, and somewhere else (maybe http://xs-dev.laptop.org/~cscott/olpc/streams/ for joyride.

Files produced

For an image called <imagename>, the files you will find inside are...

  1. fs.zip - signatures file
  2. fs.zip.md5 - the md5 checksum of fs.zip
  3. <imagename>.crc - the crc of the image
  4. <imagename>.crc.md5 - the md5 checksum of the crc of the image
  5. <imagename>.img - the image file itself
  6. <imagename>.img.md5 - the md5 checksum of the image file

If the image contains Activities and other customizations on top of a build, you will also find the following files. Note that this is rare and typically only done once a year for images shipped on G1G1 XOs.

  1. <imagename>.zip - the customization key included in the image.
  2. <imagename>.zip.md5 - the md5 checksum of the customization key included in the image.


Implementation Details

The build system we want to have consists of the following components:

                              /------\
                             | F7 yum |--------+
                             | repos  |        |
                              \------/         V
 /--------\    +--------+   +----------+   +---------+    /------\
| upstream |   | distro |   | pkg build|   |compose  |   | stable |
|    src   |-->|  SCM   |-->|  system  |-->| tools   |-->| rainbow|
|   repos  |   | (CVS)  |   |   (Koji) |   |(joyride)|   | xtest  |
 \--------/    +--------+   +----------+   +---------+    \------/
                ^  ^                           ^
 /--------\     |  |                           |
 |patches |-----+  |      + - - - - - - - +    |
 \--------/        |      |  dropbox      |    |
                   +----->|  system       |----+
 /--------\        |      | (public_rpms) |
|   olpc   |       |      + - - - - - - - +
|  sources |-------+         
|   (GIT)  |
 \--------/


|-----------| |----------------------------| |-----------------------|
   sources         package maintenance          package composition
   CVS,GIT         Koji, Mock, rpmbuild       mash, pilgrim, ChangeLogs

The one we actually have is fairly similar but the package-construction step is done partially in Koji and partially by individual package maintainers using tools like Mock and rpmbuild.


Details

Source code

Pilgrim is composed of two large and two small scripts and many configuration files.

 pilgrim              # user-interface
 pilgrim-autobuild    # implementation
 build-one            # build the current branch
 make-repos           # touch up yum repositories

The most important branch is "autobuild". This is where changes deemed suitable for all build branches should be merged. See Pilgrim Commit Policy.

The other branches - joyride, meshtest, xtest, and rainbow store branch-specific data like which yum repositories to compose into the image, which packages to pull, which activities to include, and so on.

These data are mostly recorded in

 streams.d/olpc-branch.conf                    # a few important config variables
 streams.d/olpc-development-yum-install.conf   # yum repositories to compose, package exclusions
 streams.d/olpc-development.stream             # lists of packages and activities to install

Questions:

  • How do you enumerate the available branches? I.e. if you don't want to rely on the documentation being kept up to date...
  • Where can one find the streams.d directory?