User:Quozl/fs-update-skip-unused-blocks

From OLPC
< User:Quozl
Revision as of 03:16, 28 April 2011 by Quozl (talk | contribs)
Jump to: navigation, search

Installation of an operating system on XO-1.5 and XO-1.75 laptops uses a OpenFirmware tool called fs-update.

The user downloads a .zd file from build.laptop.org, places it on a USB drive or SD card, and then uses fs-update to copy the contents of the file to the microSD card in the laptop. Once finished the laptop can be booted and operated normally.

A .zd file is an image file. The image is built by zhashfs, during olpc-os-builder. The image is usually divided into 131072 byte blocks, with each block compressed and hashed.

The blocks are usually in ascending order. A recent development by Daniel Drake (2011-04) moves block zero from the start of the file to the end, but without changing the order. This is so that an incomplete fs-update does not result in a bootable laptop.

Many of the blocks are zero filled. This is because they are unused areas of the filesystem that olpc-os-builder did not use. The blocks are very small in the .zd file, so there is no impact to download or transport of the .zd file.

OpenFirmware obediently writes these zero filled blocks during fs-update. This means for a typical laptop with 4GB microSD card, the time taken to fs-update is constant; fourteen (14) to twenty (20) minutes.

This page describes an optimisation to remove unused blocks from the .zd file, so that the fs-update time is reduced from, say, fourteen (14) to seven (7) minutes.

New Builds

Things to do:

  • testing, using mc.zd a Tiny Core Linux build, 14Mb download, 64Mb install, takes about five seconds for fs-update, [done],
  • olpc-os-builder, merge zhashfs changes that skip patterned blocks, see patch v2, maintainer desires rewrite,
  • olpc-os-builder, merge sd card image scripts that create patterned blocks, see patch, no response,
  • openfirmware, gain agreement on meaning of zblocks: #eblocks parameter [done], correct regression in q3a64 regarding use of the parameter [done], see patch, included in q3a64q, committed in -r2192 [done], to be released in q3a65, [todo]
  • openfirmware, related problem, <trac>10825</trac>, -r2190 causes data timeouts on certain cards, investigated, solved in -r2192, [done],
  • find or create a trac ticket for fs-update speedup,
  • test 10.1.3 os860, 11.2.0 os16,
  • test olpc-os-builder and f11 fresh local build,
  • test olpc-os-builder and f14 fresh local build,
  • test nandblaster,
  • public testing, with olpc-os-builder and f11,
  • public testing, with olpc-os-builder and f14,

Existing Builds

How to remove the unused blocks from a .zd file.

This procedure takes a .zd file such as os860 from 10.1.3, and creates a new .zd file that has the unused blocks removed, which results in decreased installation time. In my tests, installation time reduces from 14 minutes to 7 minutes on a 4GB microSD.

Works on Q3A62. Generates a minor warning on Q3A63 and Q3A64, but there is no missing data. Q3A65 may have the fix.

Pristine Install

Select the .zd file you wish to process. In this example, I use os860.zd.

Use fs-update to install the file on a laptop, but do not boot from it: 

fs-update u:\os860.zd

Imaging

Boot Tiny Core Linux from USB on the laptop, using a USB drive with at least 4GB free space.

Copy the microSD card to the USB drive as an image: 

mount /dev/sda1 /mnt
dd if=/dev/mmcblk0 of=/mnt/fs.img bs=131072 count=29489
umount /mnt

(the block count is the decimal conversion of the hexadecimal 7331 found in the first line of the os860.zd file).

Pattern Fill Unused Blocks

On a high-speed system, mount the USB drive.

Create a pseudo-random pattern fill, which is the same size as the filesystem block size: 

img=fs.img
dd if=/dev/urandom of=$img.fill bs=4096 count=1 2>/dev/null

Obtain the partition offsets in bytes: 

parted $img "unit B print"

In my test, the output was: 

Number  Start      End          Size         Type     File system  Flags
 1      4194304B   71303167B    67108864B    primary  ext2         boot
 2      71303168B  3865051135B  3793747968B  primary  ext3

Setup loop block devices for the partitions within the image: 

losetup -o 4194304 /dev/loop0 $img
losetup -o 71303168B /dev/loop1 $img

Fill free blocks in the filesystems with the pseudo-random pattern: 

fill-free-blocks -v /dev/loop0 $img.fill
fill-free-blocks -v /dev/loop1 $img.fill

Disassociate the loop block devices: 

losetup -d /dev/loop0
losetup -d /dev/loop1

(requires fill-free-blocks.c, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks).

Generate .zd File

Regenerate the .zd file, skipping all blocks with the pattern: 

zhashfs 0x20000 sha256 $img /tmp/fs.zsp /tmp/fs.zd

(requires a patched version of zhashfs, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks).

The .zd file is almost the same size, but will install faster. The block display by OpenFirmware will skip forward over unused blocks.

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=User:Quozl/fs-update-skip-unused-blocks&oldid=255640"