OS Builder/Run on XO hardware: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
 
(35 intermediate revisions by 3 users not shown)
Line 1: Line 1:
It is usually possible, if a bit slow, to run OS builder on XO hardware. On platforms where fast machines are available (such as x86), using a server-class "builder" machine is recommended.
It is usually possible, if a bit slow, to run OS builder on XO hardware. On platforms where fast machines are available (such as x86), using a server-class "builder" machine is recommended.


When resources are limited, or if server-class machines are not available for the platform (such as ARM), this recipe is recommended.
When resources are limited, or if server-class machines are not easily available for the platform (such as ARM), this recipe is recommended.


The procedure has been tested with XO-1.5, XO-1.75 and XO-4 hardware and 13.2.0-13.
The procedure has been tested with XO-1.5, XO-1.75 and XO-4 hardware and 13.2.0-13. (OLPC uses this procedure on XO-4 with 13.2.7, so there's a good chance you'll face no extra difficulties using 13.2.7.)


= Ingredients=
= Ingredients =


Each of these must be dedicated to the task for as long as you need to make builds:
* an XO-1.5, XO-1.75 or XO-4,

* a SD card for [[Swap]]; minimum size 512 MB, good quality, fast access, e.g. class 10,
* an XO-1.5, XO-1.75 or XO-4<ref>use an XO-1.5 to make XO-1 or XO-1.5 builds, and use an XO-1.75 or XO-4 to make XO-1.75 or XO-4 builds. An XO-1 with 13.2.0 cannot be used because there is not enough internal storage space to install the packages described here.</ref>,
* a USB hard disk drive for storing RPMs and build files; minimum size 120 GB, fast, large capacity,
* optional<ref>we have tested without swap on XO-1.5, XO-1.75 and XO-4, each of which have at least 1GB of RAM.</ref>; an SD card for [[Swap|swap]]; minimum size 512 MB, good quality, fast access, e.g. class 10,
* a USB hard disk drive for storing RPMs and build files; minimum size 16 GB, fast, large capacity,
* an unfiltered internet connection<ref>the builder must be able to download files from repositories, and filtering may prevent file downloads, in particular one user reported RPMs starting with the name ''tracker'' were blocked.</ref>,
* unsecure laptops to test your builds with,
* a way to distribute your builds to other people.


= Recipe =
= Recipe =


Each step must be done in order:
* install 13.2.0-13,

* optional; insert a USB ethernet adapter and connect to your network,
* mark the system as a builder, so that it is not taken away by accident,
* boot to Sugar or Gnome, log in,
* install [[Release_notes/13.2.0|13.2.0-13]],<!-- or [[Release_notes/13.2.2|13.2.2-15]] on XO-4, -->
* optional, if you have no USB ethernet adapter; connect to your wireless network,
* boot to Sugar or Gnome, log in, and connect to your wireless network,
* disable automatic power management from the Sugar control panel, or in Gnome:
* optional; insert a USB ethernet adapter and connect to your wired network<ref>using a USB ethernet adapter instead of a wireless network can make builds go a bit faster, because network packets are not lost. On an XO-4 with the 8787 combined wireless and bluetooth module, use a USB ethernet adapter or upgrade the kernel, see <trac>12694</trac>. Note sure if you have an 8787? See [[Firmware/Identifying_Wireless_LAN_Device|how to check]].</ref>,
sudo touch /etc/powerd/flags/inhibit-suspend
* start a Terminal and become [[root]]:
* optional; insert external SD card, and prepare for swap -- in a terminal, as root
su
# see what block device id is the external SD, usually mmcblk1p1
* disable automatic power management<ref>automatic power management should be disabled because it interferes with the builder; the USB hard drive may not be remounted after a suspend and resume.</ref>:
mount | grep mmcblk
touch /etc/powerd/flags/inhibit-suspend
* configure the system to keep documentation after package installation<ref>the change to macros.imgcreate is needed so that the example files in olpc-os-builder are kept, which are classified as documentation for the olpc-os-builder package, and are deleted by default on OLPC OS.</ref>:
# unmount any existing partitions
sed -i '/excludedocs/d' /etc/rpm/macros.imgcreate
umount /dev/mmcblk1p1
* install some packages<ref>we need to install many packages to use the system as a builder, because by design OLPC OS does not have these packages already installed.</ref>:
yum install -y ntpdate gparted olpc-os-builder binutils git gcc make libtomcrypt-devel zlib-devel
# delete manufacturer's partition
* ensure the system date and time are correct:
# create a new partition, of at least 2GB, type 82
ntpdate pool.ntp.org
parted /dev/mmcblk1
* optional; insert an SD card, and use gparted (Applications -> System Tools -> GParted Partition Editor) to delete the manufacturer's partition and add a ''linux-swap'' partition, then add it to the filesystem table:
(parted) rm 1
eval $(blkid -o export /dev/mmcblk1p1) # get the UUID
(parted) mkpart primary linux-swap 1M -1s
echo UUID=$UUID swap swap defaults 0 0 >> /etc/fstab # add to table
(parted) quit
swapon -a -v # enable swap
* prepare the USB hard drive using gparted (Applications -> System Tools -> GParted Partition Editor), partitioning it with a single ''ext4'' partition, then add it to the filesystem table:
# prepare the new partition to be swap
eval $(blkid -o export /dev/sda1) # get the UUID
mkswap /dev/mmcblk1p1
umount $DEVNAME # unmount from /media
mkdir /oob # create a mount point
# the above will report a UUID identifier, such as
echo UUID=$UUID /oob $TYPE auto,rw 0 0 >> /etc/fstab # add to table
Setting up swapspace version 1, size = ... KiB
mount -a # mount the filesystem
no label, UUID="ff1bfee8-4051-4d53-ba5e-e21559fc3c24"
* download v7.0<ref>the v7.0 branch of olpc-os-builder is what was used to create 13.2.1.</ref> of olpc-os-builder to the USB hard drive:
cd /oob # change directory
# add this UUID to fstab to use swap automatically on every boot
git clone -b v7.0 git://dev.laptop.org/projects/olpc-os-builder # download
echo UUID=ff1bfee8-4051-4d53-ba5e-e21559fc3c24 swap swap defaults 0 0 >> /etc/fstab
cd olpc-os-builder # change directory
* build the zhashfs binary:
# enable all swap partitions in fstab
swapon -a -v
(cd bin && make)
* optional, for initial testing, not final release; customise your build to make it build faster, by editing examples/olpc-os-13.2.1-xo4.ini
* prepare the USB hard drive, partitioning it with a large partition of type "83", then use mkfs to create an ext3 or ext4 partition,
** in the ''[base]'' module, set ''versioned_fs=0'' and ''make_tree_tarball=0'', (this disables the use of ''olpc-update''),
* ensure the system date is correct, set it if necessary:
** remove the ''[usb_update]'' module, (this disables the use of ''olpc-update'' with USB drives),
date --utc --set="{{CURRENTYEAR}}-{{CURRENTMONTH}}-{{CURRENTDAY}} {{CURRENTHOUR}}:{{CURRENTMINUTE}}:{{CURRENTSECOND}}"
* install the ''olpc-os-builder'' and ''binutils'' packages:
** reduce the ''[global] langs='' to the languages you need,
** '''Note that these configuration changes disable some functionality which you might need later.''' So you should only make these changes permanently if you understand what these options do, and you know that you do not need the relevant functionality.
sed -i '/excludedocs/d' /etc/rpm/macros.imgcreate
yum install olpc-os-builder binutils
* during test builds, generating only one image is faster -- edit examples/olpc-os-13.2.1-xo4.ini to comment out the "usb_update" module.


Your setup is ready, you can now make the first build:
Your setup is ready, you can now make the first build:
sudo ./osbuilder.py examples/olpc-os-13.2.1-xo4.ini
./osbuilder.py examples/olpc-os-13.2.1-xo4.ini


The first build will take much longer than subsequent builds, because it will download RPMs and activities to disk.
The first build will take much longer than subsequent builds, because it will download RPMs and activities to disk. Subsequent builds will still download some package lists, but not as much as the first build.

After the build, the directory /oob/olpc-os-builder/build/output should have the files that can be used to install the build on another system.


= Experiences =
= Experiences =


On XO-1.5, XO-1.75 and XO-4 with a typical USB hard drive, the build takes between half an hour and an hour on subsequent runs. See [[/Experiences|experiences]] for further detail.
== 2013-07-29, XO-4, 2GB RAM, 4GB SWAP, 500GB USB HDD, 13.2.0 ==


= Decommissioning =
Build took 31 minutes.


When the builder is no longer required, shut it down, remove the SD card and repartition it, remove the USB drive and repartition it, and then reinstall OLPC OS on the system.
# time ./osbuilder.py examples/olpc-os-13.2.0-xo4.ini
...
= Notes =
real 30m55.275s
<references/>
user 12m42.790s
sys 2m8.370s

Configuration file: http://dev.laptop.org/~quozl/z/1V3fiW.txt

Hardware performance measurements:
* internal eMMC, sequential read, 49 MiB/s, (operating system),
* internal eMMC, filesystem write, 13.7 MiB/s,
* USB HDD, sequential read, 27 MiB/s, (osbuilder filesystem),
* USB HDD, filesystem write, 27 MiB/s,
* external SD, sequential read, 15 MiB/s, (swap),

Performance tweaks:
* versioned_fs=0
* make_tree_tarball=0
* no sugar
* a warm file cache (timing of a subsequent run, not an initial run)

== 2013-07-30, XO-1.75, 1GB RAM, no SWAP, 500GB USB HDD, 13.2.0 ==

Build took 43 minutes.

# time ./osbuilder.py examples/olpc-os-13.2.0-xo1.75.ini
...
real 43m13.025s
user 23m44.010s
sys 4m17.620s

Configuration file: http://dev.laptop.org/~quozl/z/1V3xjq.txt

Hardware performance measurements:
* internal eMMC, sequential read, 27.8 MiB/s, (operating system),
* internal eMMC, filesystem write, 7.38 MiB/s,
* USB HDD, sequential read, 27.5 MiB/s, (osbuilder filesystem),
* USB HDD, filesystem write, 27 MiB/s,

Performance tweaks:
* versioned_fs=0
* make_tree_tarball=0
* no sugar
* a warm file cache (timing of a subsequent run, not an initial run)

== 2013-07-30, XO-1.5, 1GB RAM, no SWAP, 500GB USB HDD, 13.2.0 ==

Build took 39 minutes.

# time ./osbuilder.py examples/olpc-os-13.2.0-xo4.ini
...
real 38m50.632s
user 21m16.550s
sys 2m58.560s

Configuration file: http://dev.laptop.org/~quozl/z/1V3vxy.txt

Hardware performance measurements:
* internal microSD, sequential read, 18.2 MiB/s, (operating system),
* internal microSD, filesystem write, 6.23 MiB/s,
* USB HDD, sequential read, 32 MiB/s, (osbuilder filesystem),
* USB HDD, filesystem write, 29.4 MiB/s,


= Fedora 20 Notes =
Performance tweaks:
* use ''modprobe loop'' before build,
* versioned_fs=0
* on XO-4, <trac>12837</trac> (yum crash, infrequent 25%) presents a difficulty.
* make_tree_tarball=0
* no sugar
* a warm file cache (timing of a subsequent run, not an initial run)

Latest revision as of 10:42, 17 September 2016

It is usually possible, if a bit slow, to run OS builder on XO hardware. On platforms where fast machines are available (such as x86), using a server-class "builder" machine is recommended.

When resources are limited, or if server-class machines are not easily available for the platform (such as ARM), this recipe is recommended.

The procedure has been tested with XO-1.5, XO-1.75 and XO-4 hardware and 13.2.0-13. (OLPC uses this procedure on XO-4 with 13.2.7, so there's a good chance you'll face no extra difficulties using 13.2.7.)

Ingredients

Each of these must be dedicated to the task for as long as you need to make builds:

  • an XO-1.5, XO-1.75 or XO-4<ref>use an XO-1.5 to make XO-1 or XO-1.5 builds, and use an XO-1.75 or XO-4 to make XO-1.75 or XO-4 builds. An XO-1 with 13.2.0 cannot be used because there is not enough internal storage space to install the packages described here.</ref>,
  • optional<ref>we have tested without swap on XO-1.5, XO-1.75 and XO-4, each of which have at least 1GB of RAM.</ref>; an SD card for swap; minimum size 512 MB, good quality, fast access, e.g. class 10,
  • a USB hard disk drive for storing RPMs and build files; minimum size 16 GB, fast, large capacity,
  • an unfiltered internet connection<ref>the builder must be able to download files from repositories, and filtering may prevent file downloads, in particular one user reported RPMs starting with the name tracker were blocked.</ref>,
  • unsecure laptops to test your builds with,
  • a way to distribute your builds to other people.

Recipe

Each step must be done in order:

  • mark the system as a builder, so that it is not taken away by accident,
  • install 13.2.0-13,
  • boot to Sugar or Gnome, log in, and connect to your wireless network,
  • optional; insert a USB ethernet adapter and connect to your wired network<ref>using a USB ethernet adapter instead of a wireless network can make builds go a bit faster, because network packets are not lost. On an XO-4 with the 8787 combined wireless and bluetooth module, use a USB ethernet adapter or upgrade the kernel, see <trac>12694</trac>. Note sure if you have an 8787? See how to check.</ref>,
  • start a Terminal and become root:
 su
  • disable automatic power management<ref>automatic power management should be disabled because it interferes with the builder; the USB hard drive may not be remounted after a suspend and resume.</ref>:
 touch /etc/powerd/flags/inhibit-suspend
  • configure the system to keep documentation after package installation<ref>the change to macros.imgcreate is needed so that the example files in olpc-os-builder are kept, which are classified as documentation for the olpc-os-builder package, and are deleted by default on OLPC OS.</ref>:
 sed -i '/excludedocs/d' /etc/rpm/macros.imgcreate
  • install some packages<ref>we need to install many packages to use the system as a builder, because by design OLPC OS does not have these packages already installed.</ref>:
 yum install -y ntpdate gparted olpc-os-builder binutils git gcc make libtomcrypt-devel zlib-devel
  • ensure the system date and time are correct:
 ntpdate pool.ntp.org
  • optional; insert an SD card, and use gparted (Applications -> System Tools -> GParted Partition Editor) to delete the manufacturer's partition and add a linux-swap partition, then add it to the filesystem table:
 eval $(blkid -o export /dev/mmcblk1p1)               # get the UUID
 echo UUID=$UUID swap swap defaults 0 0 >> /etc/fstab # add to table
 swapon -a -v                                         # enable swap
  • prepare the USB hard drive using gparted (Applications -> System Tools -> GParted Partition Editor), partitioning it with a single ext4 partition, then add it to the filesystem table:
 eval $(blkid -o export /dev/sda1)                    # get the UUID
 umount $DEVNAME                                      # unmount from /media
 mkdir /oob                                           # create a mount point
 echo UUID=$UUID /oob $TYPE auto,rw 0 0 >> /etc/fstab # add to table
 mount -a                                             # mount the filesystem
  • download v7.0<ref>the v7.0 branch of olpc-os-builder is what was used to create 13.2.1.</ref> of olpc-os-builder to the USB hard drive:
 cd /oob                                                         # change directory
 git clone -b v7.0 git://dev.laptop.org/projects/olpc-os-builder # download
 cd olpc-os-builder                                              # change directory
  • build the zhashfs binary:
 (cd bin && make)
  • optional, for initial testing, not final release; customise your build to make it build faster, by editing examples/olpc-os-13.2.1-xo4.ini
    • in the [base] module, set versioned_fs=0 and make_tree_tarball=0, (this disables the use of olpc-update),
    • remove the [usb_update] module, (this disables the use of olpc-update with USB drives),
    • reduce the [global] langs= to the languages you need,
    • Note that these configuration changes disable some functionality which you might need later. So you should only make these changes permanently if you understand what these options do, and you know that you do not need the relevant functionality.

Your setup is ready, you can now make the first build:

 ./osbuilder.py examples/olpc-os-13.2.1-xo4.ini

The first build will take much longer than subsequent builds, because it will download RPMs and activities to disk. Subsequent builds will still download some package lists, but not as much as the first build.

After the build, the directory /oob/olpc-os-builder/build/output should have the files that can be used to install the build on another system.

Experiences

On XO-1.5, XO-1.75 and XO-4 with a typical USB hard drive, the build takes between half an hour and an hour on subsequent runs. See experiences for further detail.

Decommissioning

When the builder is no longer required, shut it down, remove the SD card and repartition it, remove the USB drive and repartition it, and then reinstall OLPC OS on the system.

Notes

<references/>

Fedora 20 Notes

  • use modprobe loop before build,
  • on XO-4, <trac>12837</trac> (yum crash, infrequent 25%) presents a difficulty.