Puritan: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Software}}
{{Software}}
<b><font color=red><big>For information on current-day OLPC software build systems, see [[Build system]].</big></font></b>

{{Dated}}


== Puritan ==
== Puritan ==
[http://dev.laptop.org/git/users/mstone/puritan Source code]. [http://dev.laptop.org/git?p=users/mstone/puritan;f=README;hb=ui README]. Latest version: ''ui:'' '''[http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;h=bd0fefc0b6dcae9c816505f5f2a6057e622084a9 bd0fefc0]''', ''compilation:'' '''[http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;hb=e077d7a6b654a18f8f31d721963f6964db347f47 e077d7a6]'''.
[http://dev.laptop.org/git/users/mstone/puritan Source code]. [http://dev.laptop.org/git?p=users/mstone/puritan;f=README;hb=ui README]. Latest version: ''ui:'' '''[http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;hb=e7a6bdb6dfa0b6bd067a0c9bb12c480bea09b81f e7a6bdb6]''', ''compilation:'' '''[http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;hb=8b944b2dcdb770acf802bf7324a1d6c26bc07573 8b944b2d]'''.
{{:Puritan/Introduction}}
{{:Puritan/Introduction}}


Line 9: Line 14:


== Instructions ==
== Instructions ==
{{:Puritan/Instructions}}


To use Puritan, you need to install dependencies and to check out and run a 'ui' and a 'compilation'.

# NB: mock >= 0.9.7 is needed.
# ''debian'':
sudo apt-get install mock git-core
# ''fedora'':
sudo yum install mock git-core
(git clone git://dev.laptop.org/users/mstone/puritan compilation; cd compilation; git checkout -b 767-$USER origin/767)
git clone git://dev.laptop.org/users/mstone/puritan ui
python ui/puritan/main.py # read help
sudo python2.5 ui/puritan/main.py -v ./compilation HEAD ./results build

Some puritan compilations now feature both 'download' operations:

sudo python2.5 ui/puritan/main.py -v ./compilation HEAD ./results download

and 'interactive' error-handling:

sudo python2.5 -v ./compilation HEAD ./results build -- -i

git clone git://dev.laptop.org/users/mstone/puritan ui
/usr/bin/python2.5 ui/puritan/main.py ./compilation HEAD ./results build


== Configuration ==
== Configuration ==
Puritan is highly and easily configurable by means of small modifications to the files and directories contained in each compilation. Detailed suggestions on how to [[Puritan/Configuration|configure Puritan]] are available.


Puritan's compilations are highly and easily configurable by means of small modifications to files and directories.


=== Choosing stages and products ===
== Notes and Tricks ==


Current compilations' [http://dev.laptop.org/git?p=users/mstone/puritan;a=blob;f=main.py;h=aff7b66cfd6ea7f671fe6bd233b03c4c2436e6e3;hb=e077d7a6b654a18f8f31d721963f6964db347f47#l17 main.py] file contain a section like
* Use the [http://dev.laptop.org/git/users/mstone/puritan source], Luke! (It was written to be read!)
* with some uis and compilations, various subcommands (e.g. 'build') may fail because /etc/mock/fedora-9-i386.cfg is not present in Debian's mock package 0.9.7-2

tasks = dict(
build = [
'bootstrap',
['make_chroot', 'install_pkgs', 'install_hacks'],
'final_install',
'produce_tree',
'produce_jffs2',
'produce_tar',
'publish',
],
download = ['make_chroot', 'download_pkgs'],
run = ['run'],
)

By adding and removing entries to this datastructure, builders may customize which compilation stages will be performed in response to a request to 'build' or to 'download' or to 'run' the compiler. (New subcommands can be added in the obvious fashion.)

=== Choosing package repos ===

Current compilations' [http://dev.laptop.org/git?p=users/mstone/puritan;a=blob;f=bootstrap.py;h=aff7b66cfd6ea7f671fe6bd233b03c4c2436e6e3;hb=e077d7a6b654a18f8f31d721963f6964db347f47#l28 bootstrap.py] contains lines like

smart(['channel', '-y', '--add', '8.2-fedora', 'type=rpm-md', 'baseurl=http://mock.laptop.org/repos/koji.dist-olpc3-testing/'])

which direct the [http://labix.org/smart smart package manager] to pull packages from one of the several yum-format RPM repository located at [http://mock.laptop.org/repos/ mock.laptop.org/repos].

(The smart package manager is package- and repository-format agnostic and can process many common formats including apt-dpkg and yum-rpm.)

=== Choosing packages ===

Current compilations' package manifests are configured by means of the contents of the [http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;f=config/packages;hb=767 config/packages] directory. In order to add or remove packages from the manifest, simply touch or remove the appropriately named files in this directory.

=== Caching ===

Puritan uses technologies which can be easily configured to use caches of data. Three ways to improve puritan's caching behavior include:

* Using [http://fedoraproject.org/wiki/Extras/MockTricks#Root_cache mock buildroot caching] to conserve bandwidth when preparing buildroots.
* Configuring smart to use an http caching proxy in the compilation's [http://dev.laptop.org/git?p=users/mstone/puritan;a=blob;f=bootstrap.py;h=6a9fc3356245bae10f5fadfaede83d495cfbadf0;hb=e077d7a6b654a18f8f31d721963f6964db347f47#l41 bootstrap.py] file.
* Configuring smart to pull packages from a (manually generated) local package repository (rather than from the internet), also via [http://dev.laptop.org/git?p=users/mstone/puritan;a=blob;f=bootstrap.py;h=6a9fc3356245bae10f5fadfaede83d495cfbadf0;hb=e077d7a6b654a18f8f31d721963f6964db347f47#l34 bootstrap.py].

== Notes ==

* with some uis and compilations, the last step may fail because /etc/mock/fedora-9-i386.cfg is not present in Debian's mock package 0.9.7-2
** one workaround is to create it from http://teach.laptop.org/~mstone/fedora-9-i386.cfg.
** one workaround is to create it from http://teach.laptop.org/~mstone/fedora-9-i386.cfg.
* Mock and SElinux do not interact very well.
* Mock and SElinux do not interact very well.
** at least one workaround is discussed [http://fedoraproject.org/wiki/Extras/MockTricks#Using_mock_under_SELinux on the Fedora MockTricks]] page.
** at least one workaround is discussed [http://fedoraproject.org/wiki/Extras/MockTricks#Using_mock_under_SELinux on the Fedora MockTricks]] page.
** the author simply disables selinux with <tt>sudo setenforce 0</tt> in order to avoid the hassle.
* [http://meld.sourceforge.net/ Meld] is a great tool for diffing the filesystem trees produced by the 'produce_tree' build stage and for comparing your builds to those made by others.
* [[Git]] will be your friend, but, to get the most out of the relationship, you're going to have to invest in it. Ask for help when you need it!
** <tt>gitk --all</tt> or <tt>git log</tt> will show you lots of other compilations you could run!
* The latest compilation now automatically caches packages in a directory called 'smart/packages' inside the compilation. Some other state is also cached, so if things start behaving strangely, try deleting the 'smart' directory.

== Help Out ==


Puritan was designed from the ground up to support decentralized development because its author felt that deployment autonomy was going to be critical to the success of the OLPC project. Therefore, please use your autonomy! In particular, please help improve puritan by:
==== Help Out ====
Finally, please help improve puritan by:


* Using it! As much as you can! Produce cool disk images! Publish your patches! (patch review available on request).
* Making puritan work on your platform - it's only dependencies are python2.5, git-core, and mock!
* Making puritan work on your platform - it's only dependencies are python2.5, git-core, and mock!
* Maintaining the devel_ext3 compilation
* Maintaining the devel_ext3 compilation
* Adding some reasonable package or buildroot caching system so that it runs faster without impairing build repeatability
* Adding some reasonable ''automatic'' package or buildroot caching system so that it runs faster without impairing build repeatability
* Improving the UI with commands for manipulating compilations, or for diffing builds, or for profiling compilations, or ...
* Improving the UI with commands for manipulating compilations, or for diffing builds, or for profiling compilations, or ...


[[Category:Build system]]
[[Category:Puritan]]

Latest revision as of 16:56, 8 February 2011

For information on current-day OLPC software build systems, see Build system.


Emblem-warning.png The currency of this article or section may be limited by out-of-date information.
There may be relevant discussion on its talk page


Puritan

Source code. README. Latest version: ui: e7a6bdb6, compilation: 8b944b2d. Puritan is a disk-image compiler which converts source material including packages, activities, and hacks into installable disk images. It consists of two pieces: a UI and a family of compilations (example). The compilations are simple Python programs which populate a filesystem with the materials you supply, then format it for distribution. The UI runs the compilations in a controlled environment created by Mock and configured according to the compilation's bootstrap and dependencies files.

Puritan was developed as a replacement for Pilgrim but was never officially adopted by OLPC. For details on the active OLPC build system, see Build system.

Principles

Puritan is based on several fundamental principles.

  1. Tools should be usable for both decentralized and centralized patterns of development.
  2. People **will** want to make unpackaged changes to their builds.
  3. Caching matters.
  4. Interactive development matters.
  5. Build reproducibility matters.
  6. Error-detection, handling, and cleanup matter.
  7. People working on the OLPC build system will become accustomed to git and python.


Instructions

Preparation

To use Puritan, you need to install dependencies:

 # NB: mock >= 0.9.7 is needed.
 # debian:
 sudo apt-get install mock git-core
 # fedora:
 sudo yum install mock git-core
 # both
 sudo usermod -a -G mock $USER
 newgrp mock

and to check out and run a 'ui' and a 'compilation'.

 (git clone git://dev.laptop.org/users/mstone/puritan compilation; cd compilation; git branch --track 767 origin/767; git checkout 767)
 git clone git://dev.laptop.org/users/mstone/puritan ui
 python2.5 ui/puritan/main.py   # read help
 sudo python2.5 ui/puritan/main.py -v build ./compilation HEAD ./results -- -i


Interactive Debugging

The '-i' option above enables an interactive error-handling mode. If an error occurs,

with the most recent compilation and ui, you will be dropped directly into bash inside the buildroot.

  • to resume the compilation, type
 exit 0
  • to quit the compilation, type
 exit 1

with earlier compilations which still support interactive error-handling: instructions


Other Notes

Fedora

Note: if you encounter permission errors when running this command, you may need to:

 sudo setenforce 0  # disable selinux

Note: a word of warning: while _my_ uis and compilations will treat your computer nicely, other people's might not. Caveat emptor.

Ubuntu

mock_0.9.7-2 on ubuntu requires [1] in order to run. Additionally, if you run puritan as root, you should either

usermod -a -G mock -u $SUDO_UID
unset SUDO_UID  # or
All

Some puritan compilations now feature a 'download' operation which can be used to download all packages that the compilation will install for later (offline) use:

 sudo python2.5 ui/puritan/main.py -v download ./compilation HEAD ./results 

This command will generate a program in the results directory which can be used to download the packages.


Configuration

Puritan is highly and easily configurable by means of small modifications to the files and directories contained in each compilation. Detailed suggestions on how to configure Puritan are available.


Notes and Tricks

  • Use the source, Luke! (It was written to be read!)
  • with some uis and compilations, various subcommands (e.g. 'build') may fail because /etc/mock/fedora-9-i386.cfg is not present in Debian's mock package 0.9.7-2
  • Mock and SElinux do not interact very well.
    • at least one workaround is discussed on the Fedora MockTricks] page.
    • the author simply disables selinux with sudo setenforce 0 in order to avoid the hassle.
  • Meld is a great tool for diffing the filesystem trees produced by the 'produce_tree' build stage and for comparing your builds to those made by others.
  • Git will be your friend, but, to get the most out of the relationship, you're going to have to invest in it. Ask for help when you need it!
    • gitk --all or git log will show you lots of other compilations you could run!
  • The latest compilation now automatically caches packages in a directory called 'smart/packages' inside the compilation. Some other state is also cached, so if things start behaving strangely, try deleting the 'smart' directory.

Help Out

Puritan was designed from the ground up to support decentralized development because its author felt that deployment autonomy was going to be critical to the success of the OLPC project. Therefore, please use your autonomy! In particular, please help improve puritan by:

  • Using it! As much as you can! Produce cool disk images! Publish your patches! (patch review available on request).
  • Making puritan work on your platform - it's only dependencies are python2.5, git-core, and mock!
  • Maintaining the devel_ext3 compilation
  • Adding some reasonable automatic package or buildroot caching system so that it runs faster without impairing build repeatability
  • Improving the UI with commands for manipulating compilations, or for diffing builds, or for profiling compilations, or ...