Pilgrim: Difference between revisions

From OLPC
Jump to navigation Jump to search
(questions added in need of answers)
No edit summary
 
(10 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<b><font color=red><big>For information on current-day OLPC software build systems, see [[Build system]].</big></font></b>
{{Software}}
{{Build system}}


{{Deprecated}}
== Overview ==


Pilgrim is a disk-image compiler. This means that it converts collections of packages (RPMs) and hacks into [[Clean-install|clean-installable]] disk images for the XO. It is typically run on a dedicated Linux server or virtual machine like xs-dev.laptop.org or pilgrim.laptop.org. 2010 OLPC builds use the [[OS Builder]] utility instead.
Please add text to answer the following questions:
* what is Pilgrim?
* is it already on XO laptops or do you need to install it?
* What problem does it solve?
* Who is expected to use it?
* What are the typical use cases for Pilgrim?


== Output from the Build System ==
== Details ==

The build system produces things:

# An RPM repository, currently located under the [http://xs-dev.laptop.org/~cscott/repos repository root]
# The output of the build system is currently located under the streams root at [http://pilgrim.laptop.org/~pilgrim/olpc/streams/ http://pilgrim.laptop.org/~pilgrim/olpc/streams/] for update.1, and somewhere else (maybe [http://xs-dev.laptop.org/~cscott/olpc/streams/ http://xs-dev.laptop.org/~cscott/olpc/streams/] for joyride.

=== Files produced ===

For an image called <code><imagename></code>, the files you will find inside are...

# '''fs.zip''' - signatures file
# '''fs.zip.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of fs.zip
# '''<imagename>.crc''' - the [http://en.wikipedia.org/wiki/Cyclic_redundancy_check crc] of the image
# '''<imagename>.crc.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the [http://en.wikipedia.org/wiki/Cyclic_redundancy_check crc] of the image
# '''<imagename>.img''' - the image file itself
# '''<imagename>.img.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the image file

If the image contains Activities and other customizations on top of a build, you will also find the following files. Note that this is rare and typically only done once a year for images shipped on G1G1 XOs.

# '''<imagename>.zip''' - the [[customization key]] included in the image.
# '''<imagename>.zip.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the [[customization key]] included in the image.


== Implementation Details ==

The build system we want to have consists of the following components:

<pre>
/------\
| F7 yum |--------+
| repos | |
\------/ V
/--------\ +--------+ +----------+ +---------+ /------\
| upstream | | distro | | pkg build| |compose | | stable |
| src |-->| SCM |-->| system |-->| tools |-->| rainbow|
| repos | | (CVS) | | (Koji) | |(joyride)| | xtest |
\--------/ +--------+ +----------+ +---------+ \------/
^ ^ ^
/--------\ | | |
|patches |-----+ | + - - - - - - - + |
\--------/ | | dropbox | |
+----->| system |----+
/--------\ | | (public_rpms) |
| olpc | | + - - - - - - - +
| sources |-------+
| (GIT) |
\--------/


|-----------| |----------------------------| |-----------------------|
sources package maintenance package composition
CVS,GIT Koji, Mock, rpmbuild mash, pilgrim, ChangeLogs
</pre>

The one we actually have is fairly similar but the package-construction step is done partially in [https://fedoraproject.org/wiki/Koji Koji] and partially by individual package maintainers using tools like [https://fedoraproject.org/wiki/Projects/Mock Mock] and rpmbuild.


==== Details ====


[http://dev.laptop.org/git/users/cscott/pilgrim Source code]
[http://dev.laptop.org/git/users/cscott/pilgrim Source code]
Line 25: Line 77:


The other branches - joyride, meshtest, xtest, and rainbow store branch-specific data like which yum repositories to compose into the image, which packages to pull, which activities to include, and so on.
The other branches - joyride, meshtest, xtest, and rainbow store branch-specific data like which yum repositories to compose into the image, which packages to pull, which activities to include, and so on.



These data are mostly recorded in
These data are mostly recorded in
Line 36: Line 87:
* How do you enumerate the available branches? I.e. if you don't want to rely on the documentation being kept up to date...
* How do you enumerate the available branches? I.e. if you don't want to rely on the documentation being kept up to date...
* Where can one find the streams.d directory?
* Where can one find the streams.d directory?



== Motivation ==

This tutorial explains how you can build a completely customized OS image which can be used with:

* Emulators such as [[QEMU]] and [[VMware]]
* Laptops running in developer mode (see [[Activation and developer keys]] about running your xo in developer mode and [[Clean-install procedure]] about installing OS images)

== Prerequisites ==
The following procedure has been tested on a Fedora 8 x86_64, but should work just fine on Fedora 7 and on i386.

Any Linux distribution capable of running yum should be able to run [[Pilgrim]], maybe with minor adjustments.

The intended audience of this document is an OS developer with some experience in rebuilding RPMs and managing yum repositories.


== Preparation ==

Before installing pilgrim, you need to install mtd-utils.

sudo yum install mtd-utils

git clone git://dev.laptop.org/users/cscott/pilgrim
cd pilgrim
make
sudo make install

Note: this will litter your system with pilgrim files, but you can clean it up later with <tt>make uninstall</tt>.


== Prepare your custom repositories ==

Build your custom RPMs and place them in a directory accessible via http.
It is recommended, but not necessary, to adopt the standard Fedora layout:

i386/os/ - i386 and noarch packages
i386/debug/ - debuginfo packages
source/ - src rpms

Create or update the RPM metadata information for your repositories:

createrepo i386/os
createrepo i386/debug
createrepo source


== Pilgrim customization ==

=== Adding your yum repositories ===

Edit <tt>streams.d/olpc-development-yum-install.conf</tt> and append your yum repository
(you only need the one containing the binary RPMs here). Mine looks like this:

[olpc-bernie]
name=OLPC bernie - i386
baseurl=http://www.codewiz.org/pub/olpc-bernie/i386/os/
enabled=1
gpgcheck=0

Note that I've explicitly specified <tt>i386</tt> because the <tt>$basearch</tt> variable would expand to <tt>x86_64</tt> on my host.

If you provide the full set of OS packages in your repository, then you should remove the default repositories altogether.


=== Adding repositories for the target ===

Edit <tt>streams.d/olpc-development.stream</tt>, and find the line saying
"Writing out configuration for OLPC yum repo". The repositories listed here
will appear in <tt>/etc/yum.repos.d</tt> on the target and are only used when
the image is booted on the target.

Add your yum repositories below the official ones.

This is what I do for xtest:

# keep this in sync with same repo defined in olpc-development-yum-install.conf
cat <<EOF > $INSTALL_ROOT/etc/yum.repos.d/olpc-development.repo
[olpc_development]
name=OLPC development
baseurl=http://koji.fedoraproject.org/static-repos/dist-olpc2-build-current/i386/
enabled=1
gpgcheck=0
EOF
cat <<EOF > $INSTALL_ROOT/etc/yum.repos.d/olpc-bernie.repo
[olpc-bernie]
name=OLPC bernie - \$basearch
baseurl=http://www.codewiz.org/pub/olpc-bernie/\$basearch/os/
enabled=1
gpgcheck=0
[olpc-bernie-debuginfo]
name=OLPC bernie - \$basearch - Debug
baseurl=http://www.codewiz.org/pub/olpc-bernie/\$basearch/debug/
enabled=0
gpgcheck=0
[olpc-bernie-source]
name=OLPC bernie - Source
baseurl=http://www.codewiz.org/pub/olpc-bernie/source/
enabled=0
gpgcheck=0
EOF

=== Adding and removing packages ===

The list of packages that will be installed is in <tt>streams.d/olpc-development.stream</tt>.

If your spec files are correct, you will only need to list the leaf packages here and yum will pull in all their dependencies automatically.


== Further reading ==

Read <tt>README.olpc</tt> in the root directory of Pilgrim for further information.

The short story is:

(become root)
./pilgrim-autobuild --config-dir . --stream olpc-development --dest-dir .

If you're only interest in the jffs2 image, do:

./pilgrim-autobuild --config-dir . --stream olpc-development --dest-dir . --variant devel_jffs2

To install OS images see [[Clean-install procedure]].

{{Software}}
[[Category:Build system]]

Latest revision as of 16:50, 8 February 2011

For information on current-day OLPC software build systems, see Build system.

Stop hand.png WARNING:
The content of this section is considered
DEPRECATED and OBSOLETE
It is preserved for historical or documenting reasons.

Pilgrim is a disk-image compiler. This means that it converts collections of packages (RPMs) and hacks into clean-installable disk images for the XO. It is typically run on a dedicated Linux server or virtual machine like xs-dev.laptop.org or pilgrim.laptop.org. 2010 OLPC builds use the OS Builder utility instead.

Output from the Build System

The build system produces things:

  1. An RPM repository, currently located under the repository root
  2. The output of the build system is currently located under the streams root at http://pilgrim.laptop.org/~pilgrim/olpc/streams/ for update.1, and somewhere else (maybe http://xs-dev.laptop.org/~cscott/olpc/streams/ for joyride.

Files produced

For an image called <imagename>, the files you will find inside are...

  1. fs.zip - signatures file
  2. fs.zip.md5 - the md5 checksum of fs.zip
  3. <imagename>.crc - the crc of the image
  4. <imagename>.crc.md5 - the md5 checksum of the crc of the image
  5. <imagename>.img - the image file itself
  6. <imagename>.img.md5 - the md5 checksum of the image file

If the image contains Activities and other customizations on top of a build, you will also find the following files. Note that this is rare and typically only done once a year for images shipped on G1G1 XOs.

  1. <imagename>.zip - the customization key included in the image.
  2. <imagename>.zip.md5 - the md5 checksum of the customization key included in the image.


Implementation Details

The build system we want to have consists of the following components:

                              /------\
                             | F7 yum |--------+
                             | repos  |        |
                              \------/         V
 /--------\    +--------+   +----------+   +---------+    /------\
| upstream |   | distro |   | pkg build|   |compose  |   | stable |
|    src   |-->|  SCM   |-->|  system  |-->| tools   |-->| rainbow|
|   repos  |   | (CVS)  |   |   (Koji) |   |(joyride)|   | xtest  |
 \--------/    +--------+   +----------+   +---------+    \------/
                ^  ^                           ^
 /--------\     |  |                           |
 |patches |-----+  |      + - - - - - - - +    |
 \--------/        |      |  dropbox      |    |
                   +----->|  system       |----+
 /--------\        |      | (public_rpms) |
|   olpc   |       |      + - - - - - - - +
|  sources |-------+         
|   (GIT)  |
 \--------/


|-----------| |----------------------------| |-----------------------|
   sources         package maintenance          package composition
   CVS,GIT         Koji, Mock, rpmbuild       mash, pilgrim, ChangeLogs

The one we actually have is fairly similar but the package-construction step is done partially in Koji and partially by individual package maintainers using tools like Mock and rpmbuild.


Details

Source code

Pilgrim is composed of two large and two small scripts and many configuration files.

 pilgrim              # user-interface
 pilgrim-autobuild    # implementation
 build-one            # build the current branch
 make-repos           # touch up yum repositories

The most important branch is "autobuild". This is where changes deemed suitable for all build branches should be merged. See Pilgrim Commit Policy.

The other branches - joyride, meshtest, xtest, and rainbow store branch-specific data like which yum repositories to compose into the image, which packages to pull, which activities to include, and so on.

These data are mostly recorded in

 streams.d/olpc-branch.conf                    # a few important config variables
 streams.d/olpc-development-yum-install.conf   # yum repositories to compose, package exclusions
 streams.d/olpc-development.stream             # lists of packages and activities to install

Questions:

  • How do you enumerate the available branches? I.e. if you don't want to rely on the documentation being kept up to date...
  • Where can one find the streams.d directory?


Motivation

This tutorial explains how you can build a completely customized OS image which can be used with:

Prerequisites

The following procedure has been tested on a Fedora 8 x86_64, but should work just fine on Fedora 7 and on i386.

Any Linux distribution capable of running yum should be able to run Pilgrim, maybe with minor adjustments.

The intended audience of this document is an OS developer with some experience in rebuilding RPMs and managing yum repositories.


Preparation

Before installing pilgrim, you need to install mtd-utils.

 sudo yum install mtd-utils
 git clone git://dev.laptop.org/users/cscott/pilgrim
 cd pilgrim
 make
 sudo make install

Note: this will litter your system with pilgrim files, but you can clean it up later with make uninstall.


Prepare your custom repositories

Build your custom RPMs and place them in a directory accessible via http. It is recommended, but not necessary, to adopt the standard Fedora layout:

 i386/os/     - i386 and noarch packages
 i386/debug/  - debuginfo packages
 source/      - src rpms

Create or update the RPM metadata information for your repositories:

createrepo i386/os
createrepo i386/debug
createrepo source


Pilgrim customization

Adding your yum repositories

Edit streams.d/olpc-development-yum-install.conf and append your yum repository (you only need the one containing the binary RPMs here). Mine looks like this:

[olpc-bernie]
name=OLPC bernie - i386
baseurl=http://www.codewiz.org/pub/olpc-bernie/i386/os/
enabled=1
gpgcheck=0

Note that I've explicitly specified i386 because the $basearch variable would expand to x86_64 on my host.

If you provide the full set of OS packages in your repository, then you should remove the default repositories altogether.


Adding repositories for the target

Edit streams.d/olpc-development.stream, and find the line saying "Writing out configuration for OLPC yum repo". The repositories listed here will appear in /etc/yum.repos.d on the target and are only used when the image is booted on the target.

Add your yum repositories below the official ones.

This is what I do for xtest:

   # keep this in sync with same repo defined in olpc-development-yum-install.conf
   cat <<EOF > $INSTALL_ROOT/etc/yum.repos.d/olpc-development.repo
[olpc_development]
name=OLPC development
baseurl=http://koji.fedoraproject.org/static-repos/dist-olpc2-build-current/i386/
enabled=1
gpgcheck=0
EOF

   cat <<EOF > $INSTALL_ROOT/etc/yum.repos.d/olpc-bernie.repo
[olpc-bernie]
name=OLPC bernie - \$basearch
baseurl=http://www.codewiz.org/pub/olpc-bernie/\$basearch/os/
enabled=1
gpgcheck=0

[olpc-bernie-debuginfo]
name=OLPC bernie - \$basearch - Debug
baseurl=http://www.codewiz.org/pub/olpc-bernie/\$basearch/debug/
enabled=0
gpgcheck=0

[olpc-bernie-source]
name=OLPC bernie - Source
baseurl=http://www.codewiz.org/pub/olpc-bernie/source/
enabled=0
gpgcheck=0
EOF

Adding and removing packages

The list of packages that will be installed is in streams.d/olpc-development.stream.

If your spec files are correct, you will only need to list the leaf packages here and yum will pull in all their dependencies automatically.


Further reading

Read README.olpc in the root directory of Pilgrim for further information.

The short story is:

 (become root)
 ./pilgrim-autobuild --config-dir . --stream olpc-development --dest-dir .

If you're only interest in the jffs2 image, do:

 ./pilgrim-autobuild --config-dir . --stream olpc-development --dest-dir . --variant devel_jffs2

To install OS images see Clean-install procedure.