User:Quozl/fs-update-skip-unused-blocks

Installation of an operating system on XO-1.5 and XO-1.75 laptops uses a OpenFirmware tool called fs-update.

The user downloads a .zd file from build.laptop.org, places it on a USB drive or SD card, and then uses fs-update to copy the contents of the file to the microSD card in the laptop. Once finished the laptop can be booted and operated normally.

A .zd file is an image file. The image is built by zhashfs, during olpc-os-builder. The image is usually divided into 131072 byte blocks, with each block compressed and hashed.

The blocks are usually in ascending order. A recent development by Daniel Drake (2011-04) moves block zero from the start of the file to the end, but without changing the order. This is so that an incomplete fs-update does not result in a bootable laptop.

Many of the blocks are zero filled. This is because they are unused areas of the filesystem that olpc-os-builder did not use. The blocks are very small in the .zd file, so there is no impact to download or transport of the .zd file.

OpenFirmware obediently writes these zero filled blocks during fs-update. This means for a typical laptop with 4GB microSD card, the time taken to fs-update is constant; fourteen (14) to twenty (20) minutes.

This page describes an optimisation to remove unused blocks from the .zd file, so that the fs-update time is reduced from, say, fourteen (14) to seven (7) minutes.

A .zd file with unused blocks removed is called sparse.

New Builds

Things to do [all done]:

• find or create a trac ticket for fs-update speedup, [done, <trac>10844</trac>]
• testing, using mc.zd a sparse Tiny Core Linux build, 14Mb download, 64Mb install, takes about five seconds for fs-update, [done],
• olpc-os-builder, merge zhashfs changes that skip patterned blocks in order to generate sparse .zd files, see patch v2, merged, [done],
• olpc-os-builder, merge sd card image scripts that create patterned blocks for zhashfs to recognise, see patch, merged, [done],
• openfirmware, gain agreement on meaning of zblocks: #eblocks parameter, correct regression in q3a64 regarding use of the parameter, see patch, released in Q3A65, [done]
• openfirmware, related problem, <trac>10825</trac>, -r2190 causes data timeouts on certain cards, investigated, solved in -r2192, [done],
• test whether an rsync from a sparse to a non-sparse seed file results in any substantial bandwidth saving, yes, very substantial, speedup ratio of 290 to 1, [done]
• public testing, a sparse 10.1.3 announcement, one test report, [done]
• public testing, a sparse 11.2.0 announcement, [done]
• test olpc-os-builder and f11 fresh local build, [done]
• test olpc-os-builder and f14 fresh local build, [done, 7min 37sec for 773MB .zd4],
• test with Nandblaster_for_XO-1.5, [done, slight reduction of about 7min],
• public testing, with olpc-os-builder and f11, [not done, no apparent interest]
• public testing, with olpc-os-builder and f14, [done]
• port to XO-1.75. [done, in os10]

Existing Builds

How to remove the unused blocks from a .zd file.

This procedure takes a .zd file such as os860 from 10.1.3, and creates a new .zd file that has the unused blocks removed, which results in decreased installation time. In my tests, installation time reduces from 14 minutes to 7 minutes on a 4GB microSD.

Works on Q3A62. Generates a minor warning on Q3A63 and Q3A64, but there is no missing data. Q3A65 has the fix.

Pristine Install

Select the .zd file you wish to process. In this example, I use os860.zd.

Use fs-update to install the file on a laptop, but do not boot from it:

fs-update u:\os860.zd

Imaging

Boot Tiny Core Linux from USB on the laptop, using a USB drive with at least 4GB free space.

Copy the microSD card to the USB drive as an image:

mount /dev/sda1 /mnt
dd if=/dev/mmcblk0 of=/mnt/fs.img bs=131072 count=29489
umount /mnt

(the block count is the decimal conversion of the hexadecimal 7331 found in the first line of the os860.zd file).

Pattern Fill Unused Blocks

On a high-speed system, mount the USB drive.

Create a pseudo-random pattern fill, which is the same size as the filesystem block size:

img=fs.img
dd if=/dev/urandom of=$img.fill bs=4096 count=1 2>/dev/null

Obtain the partition offsets in bytes:

parted $img "unit B print"

In my test, the output was:

Number  Start      End          Size         Type     File system  Flags
 1      4194304B   71303167B    67108864B    primary  ext2         boot
 2      71303168B  3865051135B  3793747968B  primary  ext3

Setup loop block devices for the partitions within the image:

losetup -o 4194304 /dev/loop0 $img
losetup -o 71303168 /dev/loop1 $img

Fill free blocks in the filesystems with the pseudo-random pattern:

fill-free-blocks -v /dev/loop0 $img.fill
fill-free-blocks -v /dev/loop1 $img.fill

Disassociate the loop block devices:

losetup -d /dev/loop0
losetup -d /dev/loop1

(requires fill-free-blocks.c, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks).

Generate .zd File

Regenerate the .zd file, skipping all blocks with the pattern:

zhashfs 0x20000 sha256 $img /tmp/fs.zsp /tmp/fs.zd

(requires a patched version of zhashfs, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks, or ensure you are using the change).

The .zd file is almost the same size, but will install faster. The block display by OpenFirmware will skip forward over unused blocks.

Alternatives, zdextract

zdextract in bios-crypto.git can be used to convert .zd file to .img file without using a laptop:

zdextract < os860.zd > fs.img

This method replaces the first two steps above.

Due to the patch that places block zero last, a patch to zdextract must also be used.