User:Quozl/fs-update-skip-unused-blocks
Installation of an operating system on XO-1.5 and XO-1.75 laptops uses a OpenFirmware tool called fs-update.
The user downloads a .zd file from build.laptop.org, places it on a USB drive or SD card, and then uses fs-update to copy the contents of the file to the microSD card in the laptop. Once finished the laptop can be booted and operated normally.
A .zd file is an image file. The image is built by zhashfs, during olpc-os-builder. The image is usually divided into 131072 byte blocks, with each block compressed and hashed.
The blocks are usually in ascending order. A recent development by Daniel Drake (2011-04) moves block zero from the start of the file to the end, but without changing the order. This is so that an incomplete fs-update does not result in a bootable laptop.
Many of the blocks are zero filled. This is because they are unused areas of the filesystem that olpc-os-builder did not use. The blocks are very small in the .zd file, so there is no impact to download or transport of the .zd file.
OpenFirmware obediently writes these zero filled blocks during fs-update. This means for a typical laptop with 4GB microSD card, the time taken to fs-update is constant; fourteen (14) to twenty (20) minutes.
This page describes an optimisation to remove unused blocks from the .zd file, so that the fs-update time is reduced from, say, fourteen (14) to seven (7) minutes.
A .zd file with unused blocks removed is called sparse.
New Builds
Things to do [all done]:
- find or create a trac ticket for fs-update speedup, [done, <trac>10844</trac>]
- testing, using mc.zd a sparse Tiny Core Linux build, 14Mb download, 64Mb install, takes about five seconds for fs-update, [done],
- olpc-os-builder, merge zhashfs changes that skip patterned blocks in order to generate sparse .zd files, see patch v2, merged, [done],
- olpc-os-builder, merge sd card image scripts that create patterned blocks for zhashfs to recognise, see patch, merged, [done],
- openfirmware, gain agreement on meaning of zblocks: #eblocks parameter, correct regression in q3a64 regarding use of the parameter, see patch, released in Q3A65, [done]
- openfirmware, related problem, <trac>10825</trac>, -r2190 causes data timeouts on certain cards, investigated, solved in -r2192, [done],
- test whether an rsync from a sparse to a non-sparse seed file results in any substantial bandwidth saving, yes, very substantial, speedup ratio of 290 to 1, [done]
- public testing, a sparse 10.1.3 announcement, one test report, [done]
- public testing, a sparse 11.2.0 announcement, [done]
- test olpc-os-builder and f11 fresh local build, [done]
- test olpc-os-builder and f14 fresh local build, [done, 7min 37sec for 773MB .zd4],
- test with Nandblaster_for_XO-1.5, [done, slight reduction of about 7min],
- public testing, with olpc-os-builder and f11, [not done, no apparent interest]
- public testing, with olpc-os-builder and f14, [done]
- port to XO-1.75. [done, in os10]
Existing Builds
How to remove the unused blocks from a .zd file.
This procedure takes a .zd file such as os860 from 10.1.3, and creates a new .zd file that has the unused blocks removed, which results in decreased installation time. In my tests, installation time reduces from 14 minutes to 7 minutes on a 4GB microSD.
Works on Q3A62. Generates a minor warning on Q3A63 and Q3A64, but there is no missing data. Q3A65 has the fix.
Pristine Install
Select the .zd file you wish to process. In this example, I use os860.zd.
Use fs-update to install the file on a laptop, but do not boot from it:
fs-update u:\os860.zd
Imaging
Boot Tiny Core Linux from USB on the laptop, using a USB drive with at least 4GB free space.
Copy the microSD card to the USB drive as an image:
mount /dev/sda1 /mnt dd if=/dev/mmcblk0 of=/mnt/fs.img bs=131072 count=29489 umount /mnt
(the block count is the decimal conversion of the hexadecimal 7331 found in the first line of the os860.zd file).
Pattern Fill Unused Blocks
On a high-speed system, mount the USB drive.
Create a pseudo-random pattern fill, which is the same size as the filesystem block size:
img=fs.img dd if=/dev/urandom of=$img.fill bs=4096 count=1 2>/dev/null
Obtain the partition offsets in bytes:
parted $img "unit B print"
In my test, the output was:
Number Start End Size Type File system Flags 1 4194304B 71303167B 67108864B primary ext2 boot 2 71303168B 3865051135B 3793747968B primary ext3
Setup loop block devices for the partitions within the image:
losetup -o 4194304 /dev/loop0 $img losetup -o 71303168 /dev/loop1 $img
Fill free blocks in the filesystems with the pseudo-random pattern:
fill-free-blocks -v /dev/loop0 $img.fill fill-free-blocks -v /dev/loop1 $img.fill
Disassociate the loop block devices:
losetup -d /dev/loop0 losetup -d /dev/loop1
(requires fill-free-blocks.c, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks).
Generate .zd File
Regenerate the .zd file, skipping all blocks with the pattern:
zhashfs 0x20000 sha256 $img /tmp/fs.zsp /tmp/fs.zd
(requires a patched version of zhashfs, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks, or ensure you are using the change).
The .zd file is almost the same size, but will install faster. The block display by OpenFirmware will skip forward over unused blocks.
Alternatives, zdextract
zdextract in bios-crypto.git can be used to convert .zd file to .img file without using a laptop:
zdextract < os860.zd > fs.img
This method replaces the first two steps above.
Due to the patch that places block zero last, a patch to zdextract must also be used.