Nandblaster for XO-1.5: Difference between revisions

From OLPC
Jump to navigation Jump to search
(Added section about compressor selection)
 
(15 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<noinclude>
This page will eventually document the NANDblaster modifications for XO-1.5. For now, I'm using it to record the issues and my design thoughts.
[[Category:Network]][[Category:Firmware]][[Category:Update paths]]</noinclude>
== Quick Start ==


=== Problems ===
=== XO-1.5 ===


How to install to multiple XO-1.5 using NANDblaster:
* XO-1.5 uses managed NAND (SD card that emulates a hard disk) with ext2/3/4 filesystems instead of raw NAND with JFFS2.
* Old NANDblaster has several assumptions about raw NAND.
** Assumes that the device can have bad blocks anywhere so it has to map logical block numbers to physical block numbers to work around that.
** Assumes that the actual physical block number doesn't matter except that it must be in the correct partition. So the transmitted block number sequence can be sequential and unbroken, distributing them to good physical blocks and skipping to the next partition start as necessary. For managed NAND with conventional filesystems, an image block must be placed at exactly the specified location. That number must be transmitted and can't be assumed sequential, lest you lose the ability to skip blocks.
** Implicitly assumes that the image is already compact (internal JFFS2 compression) so there is no need to reduce network bandwidth with additional compression. For non-internally-compressed filesystems, we probably want to add some explicit compression.
** Assumes that it is possible and necessary to erase a block before writing it. Managed NAND has no concept of erase - you can write zeros or ffs, but that is not "special" at the high-level interface.
** Assumes that only the non-empty blocks will be transmitted and that the other ones should be left in the erased state. For a conventional filesystem, we need to be explicit about everything.


The build is installed to the internal microSD device. You will need a [[USB drive]] of at least 1Gb capacity, but only for the duration of the installation.
=== Ideas ===


* Prepare the USB drive:
* Probably should compress at the packet level - just prior to sending a network packet - and decompress upon reception. That minimizes the impact on the existing code, which is fairly rigid in its assumptions about how to reconstruct a full erase block from any N subblock packets. As an optimization, use a chunk size that is likely to compress to less than half an ethernet packet and send two in one packet?
** download the os*.zd file for the build you wish to install,
* Probably should add something to the protocol to represent all-0 or all-ff blocks. The former is particularly common and deserves a fast-path that conserves both network bandwidth and FEC-decoding time.
** copy the file to the USB drive,
* Even better, specify that a range of block numbers contains all zeros or maybe all <some 32-bit number>.
** rename the file to fs.zd,
* Packets specifying a range of constant-data blocks need to be transmitted redundantly to guard against packet loss.
** check that the USB drive contains the file fs.zd,
* Since the partition map is just a block, we could *nearly* ignore partition specifications in the prototol - but it might be nice to have a mode whereby you update just the contents of a single partition, after checking that the existing partition map has a corresponding partition that is large enough.
* Transmit sources:
** Clone existing image
** .zd format (necessary for secure updates because you have to sign the .zsp file)
** Preformatted packet sequence for ultimate speed.
* To reduce wear on the device, do we want to capture sections (e.g. 400 MB worth) in memory and write to disk once? Possibly increases transmit time unless we can overlap reception of additional packets with write processing. To do the overlap thing, we would divide memory in two (or maybe asymmetricly based on relative speeds of packet transmission vs FEC processing and write times). If there is enough memory for the whole thing, just use it...
* On average, I think the FEC + write is likely to be faster than the network transmission, which maxes out at about 2.4 Mbytes/sec. So we could just have a pool of packet frames in memory. Whenever we complete a set for a block, write the block and recover the packet frames. Assuming 128Kib blocks and 1310-byte packet frames (100 packets per block), we need 820K index locations - 3.3 MB assuming 32-bit pointers - per gigabyte of transmitted filesystem data. So a 16 GB filesystem takes 50MB of index, leaving 450 to 950 MB for packet storage. Assuming that the block packets are mostly tramsmitted together so you have a high probability of getting a complete set quickly, that should work well. A typical filesystem image might fit entirely in memory on 1 GiB machines. Keep the free packet frames as a linked list so finding one is a constant time operation. No need to sort them.


* Prepare the sender laptop:
=== Wish List ===
** choose a laptop, it must be an XO-1.5, it need not have anything installed on it, and if there is a build installed on it won't be changed by the process,
** check that the laptop has firmware q3a50 <!-- for nb-update --> or later, upgrade the firmware if needed,
** unlock the laptop if it is locked,


* Start the sender laptop:
* Some sort of flexible scheme where partitions could be resized on the fly.
** insert the USB drive in the laptop,
* It would be nice if there was a way to mark a block as don't care, both at the device level and the filesystem level.
** power up to the [[Ok]] prompt,
* I am considering doing the high-level parts of this in Forth, relegating only the FEC processing and compression/decompression to C. Makes it easier to debug...
** type <tt>nb-update</tt> and press ''enter'',
** wait for it to get going, it is ready when you see "pass 1" and a slowly increasing block count ... it usually takes about 15 seconds,
** leave the USB drive in the laptop,


* Start the receiver laptops, one by one or at will if you have many people helping:
** hold down the four game keys while turning on each laptop, (you may here an interrupted startup sound, this is normal),
** watch the display until the coloured boxes begin to appear,


* If a receiver laptop does not properly start:
** ensure they have firmware q3a45 or later, upgrade the firmware if needed,
** try once more, wireless contention may have prevented association,
** move it closer to the sender, try once more, radiofrequency noise may have prevented association,


* Wait for all receiver laptops to finish. To assess the state of a laptop:
=== Compressor Selection ===
** if the left-hand wireless LED is on, and a black outline is moving over the screen, then the laptop is listening for data,
** if the blocks are mostly pink, noise is hindering reception,
** if the blocks are mostly yellow, then noise. distance or a broken antenna is preventing reception; the laptop should be moved closer to the sender,
** if the blocks are filling in green, and the storage LED is flickering, then the laptop is finished listening and is writing data to internal storage; the laptop may be moved away from the sender,


* Stop the sender laptop:
The speeds and ratios below were measured on an XO-1.5 using the indicated programs to compress and decompress the file /lib/libc-2.11.so, repeating the operation several times using /dev/null as the data sink.
** press [[File:Esc.png]] ''Escape'' key on the sender laptop,
** power it off,
** remove the USB drive.


== Starting the Sender ==
{| class="wikitable"
|-
! Program
! Compress<br>MB/sec
! Decompress<br>MB/sec
! Ratio<br>comp/orig
! Comp time for<br>1 GiB sec(min)
! Decomp time for<br>1 GiB sec(min)
! Network time at<br>2 MB/sec sec(min)
! Network time at<br>1 MB/sec sec(min)
|-
| lzma
| .25
| 6
| .28
| 4096 (68.3)
| 170 (2.8)
| 143 (2.4)
| 286 (4.8)
|-
| gzip
| 2.3
| 20
| .36
| 445 (7.4)
| 51 (.85)
| 184 (3.1)
| 368 (6.1)
|-
| lzop
| 17.0
| 48
| .50
| 60 (1)
| 21 (.35)
| 256 (4.3)
| 512 (8.5)
|}


=== NANDblasting an Unsigned Filesystem Image File ===
Analysis:


To send an unsigned filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Insert the USB drive into the sending XO and type:
The selected data size of 1 GiB assumes that all-0 blocks will be detected and transmitted very efficiently at the protocol level. The non-0 portion is about 1 GiB in the clear, compressing down to between 280K and 512K.


ok nb-update
* lzma is too slow for on-the-fly compression and would only be feasible for off-line preparation of a pre-compressed image. If network conditions are bad (low throughput) it might still be worthwhile.


=== NANDblasting a Signed Filesystem Image File ===
* gzip is marginal for on-the-fly compression, especially in conjunction with the FEC generation step (need to benchmark that on XO-1.5). It will be the rate-limiting step unless the network speed is rather slow. But it still might be the winner under typical conditions. The compression time might overlap almost perfectly with the network time on the sending side. Decompression is probably fast enough not to cause problems for the receiver.


To send a signed filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Put the corresponding signature bundle (the .zip that contains a control file and a signature file) on the USB drive as "/fs.zip". Insert the USB drive into the sending XO and type:
* lzop is the speed demon, but if the network is slow, the compression ratio penalty becomes significant. However, if gzip rate-limits transmission to say 1 MB/sec, while lzop would let it run at 2 MB/sec, then lzop would win handily (256 < 368).


ok nb-secure
It would be nice if we could take advantage of the fact that the data in the .zd file is already compressed, but I'm not sure how to fit that into the FEC scheme without a lot of surgery. I suppose we could coalesce adjacent compressed blocks of data until we get 128K, then unblock later. That could get fairly complicated...

=== Cloning the sender's Filesystem ===

Cloning the sender's filesystem (nb-clone) is not available on XO-1.5. See [[Imaging_for_XO-1.5]] for an alternative.

=== Stopping ===

You can stop the sender by typing the ESC key or by powering off. If you type any other key, the sender will pause and ask you if you want to stop. If you then type 'y', it will stop; other keys will resume the sending.

=== "No quiet channels" - Forcing the Channel ===

The sender chooses a wireless channel automatically by scanning to find the least-busy channel. If they are all busy, i.e. they all have received signal strength values exceeding some threshold, the sender tell you which one has the least signal strength and ask for confirmation before proceeding on that channel.

You can force it to use a specific channel by appending either "1", "6", or "11" to one of the commands above, for example:

ok nb-update6

or

ok nb-secure11

If you force the channel, the sender won't check if the channel is busy, it will just start sending.

=== Changing the Redundancy ===

You can change the redundancy percentage before you execute one of the send commands. For example, to set the sender redundancy to 12% (the default is 20%), type:

ok d# 12 to redundancy

The redundancy controls the number of extra error correction packets that the sender sends for each erase block worth of source data.

== Design Notes ==

[[Nandblaster_for_XO-1.5_Design_Discussion]]

Latest revision as of 00:57, 22 August 2012

Quick Start

XO-1.5

How to install to multiple XO-1.5 using NANDblaster:

The build is installed to the internal microSD device. You will need a USB drive of at least 1Gb capacity, but only for the duration of the installation.

  • Prepare the USB drive:
    • download the os*.zd file for the build you wish to install,
    • copy the file to the USB drive,
    • rename the file to fs.zd,
    • check that the USB drive contains the file fs.zd,
  • Prepare the sender laptop:
    • choose a laptop, it must be an XO-1.5, it need not have anything installed on it, and if there is a build installed on it won't be changed by the process,
    • check that the laptop has firmware q3a50 or later, upgrade the firmware if needed,
    • unlock the laptop if it is locked,
  • Start the sender laptop:
    • insert the USB drive in the laptop,
    • power up to the Ok prompt,
    • type nb-update and press enter,
    • wait for it to get going, it is ready when you see "pass 1" and a slowly increasing block count ... it usually takes about 15 seconds,
    • leave the USB drive in the laptop,
  • Start the receiver laptops, one by one or at will if you have many people helping:
    • hold down the four game keys while turning on each laptop, (you may here an interrupted startup sound, this is normal),
    • watch the display until the coloured boxes begin to appear,
  • If a receiver laptop does not properly start:
    • ensure they have firmware q3a45 or later, upgrade the firmware if needed,
    • try once more, wireless contention may have prevented association,
    • move it closer to the sender, try once more, radiofrequency noise may have prevented association,
  • Wait for all receiver laptops to finish. To assess the state of a laptop:
    • if the left-hand wireless LED is on, and a black outline is moving over the screen, then the laptop is listening for data,
    • if the blocks are mostly pink, noise is hindering reception,
    • if the blocks are mostly yellow, then noise. distance or a broken antenna is preventing reception; the laptop should be moved closer to the sender,
    • if the blocks are filling in green, and the storage LED is flickering, then the laptop is finished listening and is writing data to internal storage; the laptop may be moved away from the sender,
  • Stop the sender laptop:
    • press Esc.png Escape key on the sender laptop,
    • power it off,
    • remove the USB drive.

Starting the Sender

NANDblasting an Unsigned Filesystem Image File

To send an unsigned filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Insert the USB drive into the sending XO and type: 

 ok nb-update

NANDblasting a Signed Filesystem Image File

To send a signed filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Put the corresponding signature bundle (the .zip that contains a control file and a signature file) on the USB drive as "/fs.zip". Insert the USB drive into the sending XO and type: 

 ok nb-secure

Cloning the sender's Filesystem

Cloning the sender's filesystem (nb-clone) is not available on XO-1.5. See Imaging_for_XO-1.5 for an alternative.

Stopping

You can stop the sender by typing the ESC key or by powering off. If you type any other key, the sender will pause and ask you if you want to stop. If you then type 'y', it will stop; other keys will resume the sending.

"No quiet channels" - Forcing the Channel

The sender chooses a wireless channel automatically by scanning to find the least-busy channel. If they are all busy, i.e. they all have received signal strength values exceeding some threshold, the sender tell you which one has the least signal strength and ask for confirmation before proceeding on that channel.

You can force it to use a specific channel by appending either "1", "6", or "11" to one of the commands above, for example: 

 ok nb-update6

or 

 ok nb-secure11

If you force the channel, the sender won't check if the channel is busy, it will just start sending.

Changing the Redundancy

You can change the redundancy percentage before you execute one of the send commands. For example, to set the sender redundancy to 12% (the default is 20%), type: 

 ok d# 12 to redundancy

The redundancy controls the number of extra error correction packets that the sender sends for each erase block worth of source data.

Design Notes

Nandblaster_for_XO-1.5_Design_Discussion

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=Nandblaster_for_XO-1.5&oldid=273683"