User:Quozl/fs-update-skip-unused-blocks: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
No edit summary
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
Installation of an operating system on XO-1.5 and XO-1.75 laptops uses a OpenFirmware tool called ''fs-update''.
Installation of an operating system on XO-1.5 and XO-1.75 laptops uses a OpenFirmware tool called [[Firmware/Storage#fs-update|fs-update]].


The user downloads a .zd file from build.laptop.org, places it on a USB drive or SD card, and then uses ''fs-update'' to copy the contents of the file to the microSD card in the laptop. Once finished the laptop can be booted and operated normally.
The user downloads a .zd file from build.laptop.org, places it on a USB drive or SD card, and then uses ''fs-update'' to copy the contents of the file to the microSD card in the laptop. Once finished the laptop can be booted and operated normally.
Line 17: Line 17:
== New Builds ==
== New Builds ==


Things to do:
Things to do [all done]:
*find or create a trac ticket for fs-update speedup,
*find or create a trac ticket for fs-update speedup, [done, <trac>10844</trac>]
*testing, using [http://dev.laptop.org/~quozl/tinycore/mc.zd mc.zd] a sparse [[Tiny Core Linux]] build, 14Mb download, 64Mb install, takes about five seconds for fs-update, [done],
*testing, using [http://dev.laptop.org/~quozl/tinycore/mc.zd mc.zd] a sparse [[Tiny Core Linux]] build, 14Mb download, 64Mb install, takes about five seconds for fs-update, [done],
*olpc-os-builder, merge zhashfs changes that skip patterned blocks in order to generate sparse .zd files, see [http://lists.laptop.org/pipermail/devel/2011-April/031855.html patch v2], maintainer desires rewrite using FIEMAP ioctl, consensus not reached,
*olpc-os-builder, merge zhashfs changes that skip patterned blocks in order to generate sparse .zd files, see [http://lists.laptop.org/pipermail/devel/2011-April/031855.html patch v2], [http://dev.laptop.org/git/projects/olpc-os-builder/commit/?id=a17b54443ecceea57df5dfc05145da2d085e2fba merged], [done],
*olpc-os-builder, merge sd card image scripts that create patterned blocks for zhashfs to recognise, see [http://lists.laptop.org/pipermail/devel/2011-April/031853.html patch], no response,
*olpc-os-builder, merge sd card image scripts that create patterned blocks for zhashfs to recognise, see [http://lists.laptop.org/pipermail/devel/2011-April/031853.html patch], [http://dev.laptop.org/git/projects/olpc-os-builder/commit/?id=cd19f86df2f746f73eff7dac808dbdf24ba1bd4b merged], [done],
*openfirmware, gain agreement on meaning of zblocks: #eblocks parameter [done], correct regression in q3a64 regarding use of the parameter [done], see [http://lists.laptop.org/pipermail/devel/2011-April/031851.html patch], included in [http://dev.laptop.org/~quozl/q3a64q.rom q3a64q], committed in -r2192 [done], to be released in q3a65, [todo]
*openfirmware, gain agreement on meaning of zblocks: #eblocks parameter, correct regression in q3a64 regarding use of the parameter, see [http://lists.laptop.org/pipermail/devel/2011-April/031851.html patch], released in [[OLPC Firmware q3a65|Q3A65]], [done]
*openfirmware, related problem, <trac>10825</trac>, -r2190 causes data timeouts on certain cards, investigated, solved in -r2192, [done],
*openfirmware, related problem, <trac>10825</trac>, -r2190 causes data timeouts on certain cards, investigated, solved in -r2192, [done],
*test whether an rsync from a non-sparse to a sparse file results in any substantial bandwidth saving,
*test whether an rsync from a sparse to a non-sparse seed file results in any substantial bandwidth saving, yes, very substantial, speedup ratio of 290 to 1, [done]
*public testing, a sparse 10.1.3 [http://lists.laptop.org/pipermail/devel/2011-April/031883.html announcement], one test report, [done]
*test a sparse 10.1.3 os860 and 11.2.0 os16,
*public testing, a sparse 11.2.0 [http://lists.laptop.org/pipermail/devel/2011-May/031904.html announcement], [done]
*test olpc-os-builder and f11 fresh local build,
*test olpc-os-builder and f14 fresh local build,
*test olpc-os-builder and f11 fresh local build, [done]
*test olpc-os-builder and f14 fresh local build, [done, 7min 37sec for 773MB .zd4],
*test with [[Nandblaster_for_XO-1.5]],
*test with [[Nandblaster_for_XO-1.5]], [done, slight reduction of about 7min],
*public testing, with olpc-os-builder and f11,
*public testing, with olpc-os-builder and f14,
*public testing, with olpc-os-builder and f11, [not done, no apparent interest]
*public testing, with olpc-os-builder and f14, [done]
*port to XO-1.75. [done, in os10]


== Existing Builds ==
== Existing Builds ==
Line 38: Line 40:
This procedure takes a .zd file such as os860 from 10.1.3, and creates a new .zd file that has the unused blocks removed, which results in decreased installation time. In my tests, installation time reduces from 14 minutes to 7 minutes on a 4GB microSD.
This procedure takes a .zd file such as os860 from 10.1.3, and creates a new .zd file that has the unused blocks removed, which results in decreased installation time. In my tests, installation time reduces from 14 minutes to 7 minutes on a 4GB microSD.


Works on Q3A62. Generates a minor warning on Q3A63 and Q3A64, but there is no missing data. Q3A65 may have the fix.
Works on Q3A62. Generates a minor warning on Q3A63 and Q3A64, but there is no missing data. Q3A65 has the fix.


=== Pristine Install ===
=== Pristine Install ===
Line 82: Line 84:


losetup -o 4194304 /dev/loop0 $img
losetup -o 4194304 /dev/loop0 $img
losetup -o 71303168B /dev/loop1 $img
losetup -o 71303168 /dev/loop1 $img


Fill free blocks in the filesystems with the pseudo-random pattern:
Fill free blocks in the filesystems with the pseudo-random pattern:
Line 101: Line 103:
zhashfs 0x20000 sha256 $img /tmp/fs.zsp /tmp/fs.zd
zhashfs 0x20000 sha256 $img /tmp/fs.zsp /tmp/fs.zd


(requires a patched version of zhashfs, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks).
(requires a patched version of zhashfs, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks, or ensure you are using the [http://dev.laptop.org/git/projects/olpc-os-builder/commit/?id=a17b54443ecceea57df5dfc05145da2d085e2fba change]).


The .zd file is almost the same size, but will install faster. The block display by OpenFirmware will skip forward over unused blocks.
The .zd file is almost the same size, but will install faster. The block display by OpenFirmware will skip forward over unused blocks.

=== Alternatives, zdextract ===

zdextract in bios-crypto.git can be used to convert .zd file to .img file without using a laptop:

zdextract < os860.zd > fs.img

This method replaces the first two steps above.

Due to the [http://dev.laptop.org/git/projects/olpc-os-builder/commit/?id=653408f6cba8b712c1b090053b91cb7862773f47 patch that places block zero last], a [http://dev.laptop.org/git/bios-crypto/commit/?id=8beb7965a9d95f6928844199b5bcdf2e69585e79 patch to zdextract] must also be used.

Latest revision as of 03:55, 22 August 2013

Installation of an operating system on XO-1.5 and XO-1.75 laptops uses a OpenFirmware tool called fs-update.

The user downloads a .zd file from build.laptop.org, places it on a USB drive or SD card, and then uses fs-update to copy the contents of the file to the microSD card in the laptop. Once finished the laptop can be booted and operated normally.

A .zd file is an image file. The image is built by zhashfs, during olpc-os-builder. The image is usually divided into 131072 byte blocks, with each block compressed and hashed.

The blocks are usually in ascending order. A recent development by Daniel Drake (2011-04) moves block zero from the start of the file to the end, but without changing the order. This is so that an incomplete fs-update does not result in a bootable laptop.

Many of the blocks are zero filled. This is because they are unused areas of the filesystem that olpc-os-builder did not use. The blocks are very small in the .zd file, so there is no impact to download or transport of the .zd file.

OpenFirmware obediently writes these zero filled blocks during fs-update. This means for a typical laptop with 4GB microSD card, the time taken to fs-update is constant; fourteen (14) to twenty (20) minutes.

This page describes an optimisation to remove unused blocks from the .zd file, so that the fs-update time is reduced from, say, fourteen (14) to seven (7) minutes.

A .zd file with unused blocks removed is called sparse.

New Builds

Things to do [all done]:

  • find or create a trac ticket for fs-update speedup, [done, <trac>10844</trac>]
  • testing, using mc.zd a sparse Tiny Core Linux build, 14Mb download, 64Mb install, takes about five seconds for fs-update, [done],
  • olpc-os-builder, merge zhashfs changes that skip patterned blocks in order to generate sparse .zd files, see patch v2, merged, [done],
  • olpc-os-builder, merge sd card image scripts that create patterned blocks for zhashfs to recognise, see patch, merged, [done],
  • openfirmware, gain agreement on meaning of zblocks: #eblocks parameter, correct regression in q3a64 regarding use of the parameter, see patch, released in Q3A65, [done]
  • openfirmware, related problem, <trac>10825</trac>, -r2190 causes data timeouts on certain cards, investigated, solved in -r2192, [done],
  • test whether an rsync from a sparse to a non-sparse seed file results in any substantial bandwidth saving, yes, very substantial, speedup ratio of 290 to 1, [done]
  • public testing, a sparse 10.1.3 announcement, one test report, [done]
  • public testing, a sparse 11.2.0 announcement, [done]
  • test olpc-os-builder and f11 fresh local build, [done]
  • test olpc-os-builder and f14 fresh local build, [done, 7min 37sec for 773MB .zd4],
  • test with Nandblaster_for_XO-1.5, [done, slight reduction of about 7min],
  • public testing, with olpc-os-builder and f11, [not done, no apparent interest]
  • public testing, with olpc-os-builder and f14, [done]
  • port to XO-1.75. [done, in os10]

Existing Builds

How to remove the unused blocks from a .zd file.

This procedure takes a .zd file such as os860 from 10.1.3, and creates a new .zd file that has the unused blocks removed, which results in decreased installation time. In my tests, installation time reduces from 14 minutes to 7 minutes on a 4GB microSD.

Works on Q3A62. Generates a minor warning on Q3A63 and Q3A64, but there is no missing data. Q3A65 has the fix.

Pristine Install

Select the .zd file you wish to process. In this example, I use os860.zd.

Use fs-update to install the file on a laptop, but do not boot from it:

fs-update u:\os860.zd

Imaging

Boot Tiny Core Linux from USB on the laptop, using a USB drive with at least 4GB free space.

Copy the microSD card to the USB drive as an image:

mount /dev/sda1 /mnt
dd if=/dev/mmcblk0 of=/mnt/fs.img bs=131072 count=29489
umount /mnt

(the block count is the decimal conversion of the hexadecimal 7331 found in the first line of the os860.zd file).

Pattern Fill Unused Blocks

On a high-speed system, mount the USB drive.

Create a pseudo-random pattern fill, which is the same size as the filesystem block size:

img=fs.img
dd if=/dev/urandom of=$img.fill bs=4096 count=1 2>/dev/null

Obtain the partition offsets in bytes:

parted $img "unit B print"

In my test, the output was:

Number  Start      End          Size         Type     File system  Flags
 1      4194304B   71303167B    67108864B    primary  ext2         boot
 2      71303168B  3865051135B  3793747968B  primary  ext3

Setup loop block devices for the partitions within the image:

losetup -o 4194304 /dev/loop0 $img
losetup -o 71303168 /dev/loop1 $img

Fill free blocks in the filesystems with the pseudo-random pattern:

fill-free-blocks -v /dev/loop0 $img.fill
fill-free-blocks -v /dev/loop1 $img.fill

Disassociate the loop block devices:

losetup -d /dev/loop0
losetup -d /dev/loop1

(requires fill-free-blocks.c, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks).

Generate .zd File

Regenerate the .zd file, skipping all blocks with the pattern:

zhashfs 0x20000 sha256 $img /tmp/fs.zsp /tmp/fs.zd

(requires a patched version of zhashfs, see http://dev.laptop.org/~quozl/fs-update-skip-unused-blocks, or ensure you are using the change).

The .zd file is almost the same size, but will install faster. The block display by OpenFirmware will skip forward over unused blocks.

Alternatives, zdextract

zdextract in bios-crypto.git can be used to convert .zd file to .img file without using a laptop:

zdextract < os860.zd > fs.img

This method replaces the first two steps above.

Due to the patch that places block zero last, a patch to zdextract must also be used.