Build system: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
 
(27 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<noinclude>{{TOCright}}</noinclude>
<noinclude>{{TOCright}}</noinclude>


== Description ==
'''Notice:''' In 2010 OLPC uses the [[OS Builder]] utility for their official builds.


The OLPC XO software build system must combine software from various sources:
This page is preserved for historical reference.
{{Dated}}
== Output from the Build System ==


* unchanged packages from Fedora,
The build system produces things:
* forked packages from Fedora, where the version in Fedora is insufficient,
* OLPC packages which are not available in Fedora,
* Sugar activities, and
* hardware specific configuration.


The build system makes software images which can be installed on XOs for testing and deployment.
# An RPM repository, currently located under the [http://xs-dev.laptop.org/~cscott/repos repository root]
# The output of the build system is currently located under the streams root at [http://pilgrim.laptop.org/~pilgrim/olpc/streams/ http://pilgrim.laptop.org/~pilgrim/olpc/streams/] for update.1, and somewhere else (maybe [http://xs-dev.laptop.org/~cscott/olpc/streams/ http://xs-dev.laptop.org/~cscott/olpc/streams/] for joyride.


While OLPC produces generic software images, almost all deployments will want to add some level of customization (such as Sugar activities and software applications, default settings, etc). To allow for this, the build system can be easily installed and used by deployments and developers, allowing for easy local customisation.
There are two iterations of the build system:
* [[pilgrim]] is the current (as of December 2008) build system
* [[puritan]] has some ideas for a nicer build system for putting software on XOs
Both are described below.


=== Files produced ===
== Current build system ==


[[OS Builder]] is used for current and upcoming OLPC software releases. Compared to its predecessors, it makes extensive use of Fedora's build tools under a higher layer which allows for organisation of components and easy customization. It is also easier to install and use.
For an image called <code><imagename></code>, the files you will find inside are...


== Files produced by the build system ==
# '''fs.zip''' - signatures file
[[Category:File formats]]
# '''fs.zip.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of fs.zip
# '''<imagename>.crc''' - the [http://en.wikipedia.org/wiki/Cyclic_redundancy_check crc] of the image
# '''<imagename>.crc.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the [http://en.wikipedia.org/wiki/Cyclic_redundancy_check crc] of the image
# '''<imagename>.img''' - the image file itself
# '''<imagename>.img.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the image file


The build system produces sets of files for each targeted OLPC model, each starting with the build number such as <tt>os''883''</tt>. It also produces a representation of the resulting filesystem for the OLPC model as one or more large files, that the OLPC's [[firmware]] knows how to copy to internal storage. The [[Release notes]] for a build document this install procedure.
If the image contains Activities and other customizations on top of a build, you will also find the following files. Note that this is rare and typically only done once a year for images shipped on G1G1 XOs.


=== For users installing ===
# '''<imagename>.zip''' - the [[customization key]] included in the image.
# '''<imagename>.zip.md5''' - the [http://en.wikipedia.org/wiki/Md5 md5] [http://en.wikipedia.org/wiki/Checksum checksum] of the [[customization key]] included in the image.


; .''file''.md5
== Maintainers ==
: a cryptographic hash of the contents of the file ''file'', so that a [[Download#Verifying|download can be verified]].
; .zd2, .zd4, .zd8
: compressed disk images containing a boot and root filesystem, for XOs with eMMC, microSD, or SD card internal storage, such as [[XO-1.5]] and [[XO-1.75]]. The [[firmware]] knows how to copy this to internal storage. The trailing number indicates the size of the disk image in GB, and must be equal to or smaller than the size of the internal storage.
; .zd''N''.zsp , .zd''N''.zsp.fs.zip
: cryptographic signatures for .zd''N'' files, used by secured laptops to verify the authenticity of the files during installation.
; .uim and .onu
: a representation of the blocks of a partitioned [[UBIFS]] image for the XO-1. The firmware knows how to copy this to its built-in NAND flash memory.
; .img and .crc
: a non-partitioned JFFS2 file system image for an [[XO-1]]. The XO-1's [[firmware]] knows how to copy this to NAND Flash memory.


=== For users updating ===
1. Scott Ananian (c_scott)


; .toc and .usb
2. Dennis Gilmore (dgilmore)
: Information for performing a system update using [[olpc-update]] without requiring network access.


=== For engineers ===
3. Bernardo Innocenti (_bernie)


; .activities.txt, .libraries.txt and .packages.txt
4. Michael Stone (m_stone)
: the [[Activities]], [[Collections]], and [[Fedora]]/OLPC RPM packages that went into the build. These are useful for build regression testing.
; .files.txt.gz
: a list of the entries in the filesystem. Most web browsers can view this compressed file directly. Used for build to build regression testing.
; .tree.tar.lzma
: the filesystem, in LZMA compressed tar format. If you carefully unpack this onto a Linux partition on a USB drive or SD card and make some changes, you may be able to boot an XO from it.
; .ext3.img
: raw disk image including partition table, suitable for copying onto a USB drive, see [[OS images for USB disks]]. Not present in recent builds ... they are generated and discarded because they are not intended to boot.


== Historical build systems and their evolution ==
== Build Branches ==


[[Pilgrim]] was the first build system, developed by OLPC and Red Hat, and used for OLPC's early software releases.
Frequently, we find ourselves needing to integrate complicated changes that cut across packages. To make this easier, we make it very easy to 'branch' a build in which the integration can be performed safely, without interfering with other work. If you are trying to perform a delicate piece of integration for which it would be helpful to have a separate build, please contact one of the [[Build system#Maintainers|build system maintainers]].


[[Puritan]] was developed after pilgrim started to show signs of outgrowing its design, calling for a cleaner and leaner system. However, it was never adopted officially by OLPC.
We currently have five build branches arranged in an inheritance tree like so:


[[Image builder]] was developed as a post-processing tool to complement Pilgrim when OLPC decided to hand off part of the image customization process to deployments; it takes Pilgrim images and applies various customizations to them to produce a new output image. This started to outgrow its design when various parts of pilgrim needed to be copied into Image builder in order to be able to make more invasive customizations.
kernel <- xtest <- datastore <- sugar <- [[Rainbow|rainbow]] <- [[Joyride|joyride]].


For initial software development for the XO-1.5 laptop, a desire to move to standard Fedora build tools emerged. [http://dev.laptop.org/git/projects/fedora-xo/log/?h=xo-1.5 fedora-xo] was used to produce the first XO-1.5 software release, but the resultant build system was messy and left no systematic process for customizations by deployments. This lead to design goals for future build systems to balance use of Fedora build tools, organisation of OLPC's special requirements, and the ability for deployments to easily obtain and use the tool while making customizations to the image.
== Instructions for Use ==
Before getting started, you will need an account on dev.laptop.org (See [[Project hosting]]).


== School Server Build Tools ==
To add a completely new piece of software (either a new activity or a new package) to a build, three steps are necessary.


See [[XS Building Software]] for details.
1. Include a source code package (.xo or .src.rpm) and a binary package (.xo or .rpm) in your dropbox (which is located at ~/public_rpms/<build>/ on dev.laptop.org). You will need to create the folder, here is an example for joyride:
mkdir -p ~/public_rpms/joyride


Note: do not remove the .xpm's or .xo's from the public_rpms location, not even after they already showed up in an image: public_rpms is not a temporary place where the build systems grab them; it will grab them from that location every run.

2. Update the change log to describe the changes made by the packages you are introducing. We accumulate changelog entries for ~/public_rpms/<build>/ChangeLog and ~/public_rpms/<build>/*.changes, so you can use a separate .changes file for each major group of packages you maintain.

3. Where possible, it's best to add your package as a new dependency of an existing package, so that the reason for the inclusion of this package is clear. If this is not possible, file a bug in trac with component 'distro' and owner 'cscott' requesting that the new package or activity be added to the build.

To verify that a package (here Web-72.xo) has been picked up you can do the following:

First you can look in the [http://xs-dev.laptop.org/~cscott/repos/joyride/ repos/joyride] if there was a Web-*.xo present.
Another interesting file in the same directory is the [http://xs-dev.laptop.org/~cscott/repos/joyride/ChangeLog ChangeLog] which is the aggregate ChangeLog that describes all of the packages pulled in from every dropbox. It contains warnings emitted when some problem prevents your RPMs from being picked up.
Finally, you can read the [http://xs-dev.laptop.org/cscott/olpc/streams/joyride/latest/devel_jffs2/build.log buildlog] to check if Web-72.xo was installed.

: [[User:Bert|Bert]] made a script that aggregates changelogs as [http://dev.laptop.org/~bert/joyride-pkgs.html joyride-pkgs.html], which is useful for comparing build versions.

=== ChangeLog Format ===

The format of this ChangeLog is

<pkg-lines> (containing ".xo", ".rpm")
<detail-lines> (not containing the string ".xo", ".rpm", "--")
-- <signature> (-- is the line marker)
<pkg>
<pkg>.rpm
<detail>
-- <signature>

An example:

Etoys-61.xo
etoys-2.2.1710-1.noarch.rpm
etoys-2.2.1710-1.src.rpm
* changed activity launch to match new Sugar
* words in tiles separated by spaces not camelCase
* adds "distance to" and "bearing to" tiles
* better .pot sort order
-- Bert Freudenberg <bert@freudenbergs.de> Tue, 16 Oct 2007
squeak-vm-3.9-12olpc1.i386.rpm
squeak-vm-3.9-12olpc1.src.rpm
* big merge with trunk
* unicode key events
* fixed dbus plugin
-- Bert Freudenberg <bert@freudenbergs.de> Mon, 15 Oct 2007

=== Common Mistakes ===

You ''must'' use different time stamps for each ChangeLog entry on the -- author line.

If there are multiple authors of a single build-patch, you should list each person's contribution like so:

pkg.rpm
* change
-- author 1
pkg.rpm
* change
-- author 2

rather than like

pkg.rpm
* change
-- author 1
* change
-- author 2

until we improve the ChangeLog parser.

== Implementation Details ==

The build system we want to have consists of the following components:

<pre>
/------\
| F7 yum |--------+
| repos | |
\------/ V
/--------\ +--------+ +----------+ +---------+ /------\
| upstream | | distro | | pkg build| |compose | | stable |
| src |-->| SCM |-->| system |-->| tools |-->| rainbow|
| repos | | (CVS) | | (Koji) | |(joyride)| | xtest |
\--------/ +--------+ +----------+ +---------+ \------/
^ ^ ^
/--------\ | | |
|patches |-----+ | + - - - - - - - + |
\--------/ | | dropbox | |
+----->| system |----+
/--------\ | | (public_rpms) |
| olpc | | + - - - - - - - +
| sources |-------+
| (GIT) |
\--------/


|-----------| |----------------------------| |-----------------------|
sources package maintenance package composition
CVS,GIT Koji, Mock, rpmbuild mash, pilgrim, ChangeLogs
</pre>

The one we actually have is fairly similar but the package-construction step is done partially in [https://fedoraproject.org/wiki/Koji Koji] and partially by individual package maintainers using tools like [https://fedoraproject.org/wiki/Projects/Mock Mock] and rpmbuild.

== Moving Forward ==

* [[Build system requirements]]

* [[Build system meeting]]

* [[Bernie's build system workflow proposal]]

== Source Code: Patches Welcome ==

=== Dropbox Mechanism ===
[http://dev.laptop.org/git/users/mstone/buildtools Source code]

The dropbox mechanism is a way for people who don't have the resources to work through koji to supply us with activities and RPMs.

It is implemented in two pieces, one running on the dropbox-host and one running on the build-host. The dropbox-host simply provides an rsync daemon that will transfer packages to the builder on request.

The build-host uses the program 'collect-pkgs.py' to aggregate packages and change-entries into an RPM repository for composition by pilgrim.

''Information transcluded from the [[Pilgrim]] page'':

{{:Pilgrim}}

''Information transcluded from the [[Puritan]] page'':

=== Puritan ===

{{:Puritan/Introduction}}

To learn more, please visit [[Puritan]].

=== Image Builder ===

If you are looking for a simpler and way of making a build, please have a look at [http://wiki.laptop.org/go/Image_builder Image Builder].

Image builder is a script which allows you to take an official OS image released by OLPC, and add in your own customizations. It is primarily intended for adding activities (.xo) and content bundles (.xol), but advanced users can script other customizations too.

=== School Server Build Tools ===

See [[XS Building Software]] for details.
[[Category:Build system]]
[[Category:Build system]]
[[Category:XO software and Fedora]]

Latest revision as of 05:50, 8 December 2011

Description

The OLPC XO software build system must combine software from various sources:

  • unchanged packages from Fedora,
  • forked packages from Fedora, where the version in Fedora is insufficient,
  • OLPC packages which are not available in Fedora,
  • Sugar activities, and
  • hardware specific configuration.

The build system makes software images which can be installed on XOs for testing and deployment.

While OLPC produces generic software images, almost all deployments will want to add some level of customization (such as Sugar activities and software applications, default settings, etc). To allow for this, the build system can be easily installed and used by deployments and developers, allowing for easy local customisation.

Current build system

OS Builder is used for current and upcoming OLPC software releases. Compared to its predecessors, it makes extensive use of Fedora's build tools under a higher layer which allows for organisation of components and easy customization. It is also easier to install and use.

Files produced by the build system

The build system produces sets of files for each targeted OLPC model, each starting with the build number such as os883. It also produces a representation of the resulting filesystem for the OLPC model as one or more large files, that the OLPC's firmware knows how to copy to internal storage. The Release notes for a build document this install procedure.

For users installing

.file.md5
a cryptographic hash of the contents of the file file, so that a download can be verified.
.zd2, .zd4, .zd8
compressed disk images containing a boot and root filesystem, for XOs with eMMC, microSD, or SD card internal storage, such as XO-1.5 and XO-1.75. The firmware knows how to copy this to internal storage. The trailing number indicates the size of the disk image in GB, and must be equal to or smaller than the size of the internal storage.
.zdN.zsp , .zdN.zsp.fs.zip
cryptographic signatures for .zdN files, used by secured laptops to verify the authenticity of the files during installation.
.uim and .onu
a representation of the blocks of a partitioned UBIFS image for the XO-1. The firmware knows how to copy this to its built-in NAND flash memory.
.img and .crc
a non-partitioned JFFS2 file system image for an XO-1. The XO-1's firmware knows how to copy this to NAND Flash memory.

For users updating

.toc and .usb
Information for performing a system update using olpc-update without requiring network access.

For engineers

.activities.txt, .libraries.txt and .packages.txt
the Activities, Collections, and Fedora/OLPC RPM packages that went into the build. These are useful for build regression testing.
.files.txt.gz
a list of the entries in the filesystem. Most web browsers can view this compressed file directly. Used for build to build regression testing.
.tree.tar.lzma
the filesystem, in LZMA compressed tar format. If you carefully unpack this onto a Linux partition on a USB drive or SD card and make some changes, you may be able to boot an XO from it.
.ext3.img
raw disk image including partition table, suitable for copying onto a USB drive, see OS images for USB disks. Not present in recent builds ... they are generated and discarded because they are not intended to boot.

Historical build systems and their evolution

Pilgrim was the first build system, developed by OLPC and Red Hat, and used for OLPC's early software releases.

Puritan was developed after pilgrim started to show signs of outgrowing its design, calling for a cleaner and leaner system. However, it was never adopted officially by OLPC.

Image builder was developed as a post-processing tool to complement Pilgrim when OLPC decided to hand off part of the image customization process to deployments; it takes Pilgrim images and applies various customizations to them to produce a new output image. This started to outgrow its design when various parts of pilgrim needed to be copied into Image builder in order to be able to make more invasive customizations.

For initial software development for the XO-1.5 laptop, a desire to move to standard Fedora build tools emerged. fedora-xo was used to produce the first XO-1.5 software release, but the resultant build system was messy and left no systematic process for customizations by deployments. This lead to design goals for future build systems to balance use of Fedora build tools, organisation of OLPC's special requirements, and the ability for deployments to easily obtain and use the tool while making customizations to the image.

School Server Build Tools

See XS Building Software for details.

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=Build_system&oldid=263664"