Nandblaster for XO-1.5: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
 
(11 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<noinclude>
[[Category:Network]][[Category:Firmware]][[Category:Update paths]]</noinclude>
== Quick Start ==
== Quick Start ==


=== XO-1.5 ===
=== XO-1.5 ===


How to install to multiple XO-1.5 using nandblaster:
How to install to multiple XO-1.5 using NANDblaster:


The build is installed to the internal microSD device. You will need a [[USB drive]] of at least 1Gb capacity, but only for the duration of the installation.
* ensure all XO-1.5 have firmware q3a45 or later,
* place the os*.zd file on a USB stick,
* select one XO-1.5 to be the transmitter, (there is no change to what is installed on it),
* insert the USB stick in the transmitter laptop,
* power up to the [[Ok]] prompt,
* type <tt>nb-tx: u:\os*.zd</tt> and press enter,
* wait five seconds, (for nb-tx: to get going),
* power up the receiving laptops one by one using the four game keys down sequence,
* wait for each laptop to finish and turn off (or reboot?),
* when all have finished, turn off the transmitter.


* Prepare the USB drive:
----
** download the os*.zd file for the build you wish to install,
** copy the file to the USB drive,
** rename the file to fs.zd,
** check that the USB drive contains the file fs.zd,


* Prepare the sender laptop:
This page will eventually document the NANDblaster modifications for XO-1.5. For now, I'm using it to record the issues and my design thoughts.
** choose a laptop, it must be an XO-1.5, it need not have anything installed on it, and if there is a build installed on it won't be changed by the process,
** check that the laptop has firmware q3a50 <!-- for nb-update --> or later, upgrade the firmware if needed,
** unlock the laptop if it is locked,


* Start the sender laptop:
=== Problems ===
** insert the USB drive in the laptop,
** power up to the [[Ok]] prompt,
** type <tt>nb-update</tt> and press ''enter'',
** wait for it to get going, it is ready when you see "pass 1" and a slowly increasing block count ... it usually takes about 15 seconds,
** leave the USB drive in the laptop,


* Start the receiver laptops, one by one or at will if you have many people helping:
* XO-1.5 uses managed NAND (SD card that emulates a hard disk) with ext2/3/4 filesystems instead of raw NAND with JFFS2.
** hold down the four game keys while turning on each laptop, (you may here an interrupted startup sound, this is normal),
* Old NANDblaster has several assumptions about raw NAND.
** watch the display until the coloured boxes begin to appear,
** Assumes that the device can have bad blocks anywhere so it has to map logical block numbers to physical block numbers to work around that.
** Assumes that the actual physical block number doesn't matter except that it must be in the correct partition. So the transmitted block number sequence can be sequential and unbroken, distributing them to good physical blocks and skipping to the next partition start as necessary. For managed NAND with conventional filesystems, an image block must be placed at exactly the specified location. That number must be transmitted and can't be assumed sequential, lest you lose the ability to skip blocks.
** Implicitly assumes that the image is already compact (internal JFFS2 compression) so there is no need to reduce network bandwidth with additional compression. For non-internally-compressed filesystems, we probably want to add some explicit compression.
** Assumes that it is possible and necessary to erase a block before writing it. Managed NAND has no concept of erase - you can write zeros or ffs, but that is not "special" at the high-level interface.
** Assumes that only the non-empty blocks will be transmitted and that the other ones should be left in the erased state. For a conventional filesystem, we need to be explicit about everything.


* If a receiver laptop does not properly start:
=== Ideas ===
** ensure they have firmware q3a45 or later, upgrade the firmware if needed,
** try once more, wireless contention may have prevented association,
** move it closer to the sender, try once more, radiofrequency noise may have prevented association,


* Wait for all receiver laptops to finish. To assess the state of a laptop:
* Probably should compress at the packet level - just prior to sending a network packet - and decompress upon reception. That minimizes the impact on the existing code, which is fairly rigid in its assumptions about how to reconstruct a full erase block from any N subblock packets. As an optimization, use a chunk size that is likely to compress to less than half an ethernet packet and send two in one packet?
** if the left-hand wireless LED is on, and a black outline is moving over the screen, then the laptop is listening for data,
* Probably should add something to the protocol to represent all-0 or all-ff blocks. The former is particularly common and deserves a fast-path that conserves both network bandwidth and FEC-decoding time.
** if the blocks are mostly pink, noise is hindering reception,
* Even better, specify that a range of block numbers contains all zeros or maybe all <some 32-bit number>.
** if the blocks are mostly yellow, then noise. distance or a broken antenna is preventing reception; the laptop should be moved closer to the sender,
* Packets specifying a range of constant-data blocks need to be transmitted redundantly to guard against packet loss.
** if the blocks are filling in green, and the storage LED is flickering, then the laptop is finished listening and is writing data to internal storage; the laptop may be moved away from the sender,
* Since the partition map is just a block, we could *nearly* ignore partition specifications in the prototol - but it might be nice to have a mode whereby you update just the contents of a single partition, after checking that the existing partition map has a corresponding partition that is large enough.
* Transmit sources:
** Clone existing image
** .zd format (necessary for secure updates because you have to sign the .zsp file)
** Preformatted packet sequence for ultimate speed.
* To reduce wear on the device, do we want to capture sections (e.g. 400 MB worth) in memory and write to disk once? Possibly increases transmit time unless we can overlap reception of additional packets with write processing. To do the overlap thing, we would divide memory in two (or maybe asymmetricly based on relative speeds of packet transmission vs FEC processing and write times). If there is enough memory for the whole thing, just use it...
* On average, I think the FEC + write is likely to be faster than the network transmission, which maxes out at about 2.4 Mbytes/sec. So we could just have a pool of packet frames in memory. Whenever we complete a set for a block, write the block and recover the packet frames. Assuming 128Kib blocks and 1310-byte packet frames (100 packets per block), we need 820K index locations - 3.3 MB assuming 32-bit pointers - per gigabyte of transmitted filesystem data. So a 16 GB filesystem takes 50MB of index, leaving 450 to 950 MB for packet storage. Assuming that the block packets are mostly transmitted together so you have a high probability of getting a complete set quickly, that should work well. A typical filesystem image might fit entirely in memory on 1 GiB machines. Keep the free packet frames as a linked list so finding one is a constant time operation. No need to sort them.


* Stop the sender laptop:
=== Wish List ===
** press [[File:Esc.png]] ''Escape'' key on the sender laptop,
** power it off,
** remove the USB drive.


== Starting the Sender ==
* Some sort of flexible scheme where partitions could be resized on the fly.
* It would be nice if there was a way to mark a block as don't care, both at the device level and the filesystem level.
* I am considering doing the high-level parts of this in Forth, relegating only the FEC processing and compression/decompression to C. Makes it easier to debug...


=== NANDblasting an Unsigned Filesystem Image File ===


To send an unsigned filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Insert the USB drive into the sending XO and type:


ok nb-update
=== Compressor Selection ===


=== NANDblasting a Signed Filesystem Image File ===
The speeds and ratios below were measured on an XO-1.5 using the indicated programs to compress and decompress the file /lib/libc-2.11.so, repeating the operation several times using /dev/null as the data sink.


To send a signed filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Put the corresponding signature bundle (the .zip that contains a control file and a signature file) on the USB drive as "/fs.zip". Insert the USB drive into the sending XO and type:
{| class="wikitable"
|-
! Program
! Compress<br>MB/sec
! Decompress<br>MB/sec
! Ratio<br>comp/orig
! Comp time for<br>1 GiB sec(min)
! Decomp time for<br>1 GiB sec(min)
! Network time at<br>2 MB/sec sec(min)
! Network time at<br>1 MB/sec sec(min)
|-
| lzma
| .25
| 6
| .28
| 4096 (68.3)
| 170 (2.8)
| 143 (2.4)
| 286 (4.8)
|-
| gzip
| 2.3
| 20
| .36
| 445 (7.4)
| 51 (.85)
| 184 (3.1)
| 368 (6.1)
|-
| lzop
| 17.0
| 48
| .50
| 60 (1)
| 21 (.35)
| 256 (4.3)
| 512 (8.5)
|}


ok nb-secure
Analysis:


=== Cloning the sender's Filesystem ===
The selected data size of 1 GiB assumes that all-0 blocks will be detected and transmitted very efficiently at the protocol level. The non-0 portion is about 1 GiB in the clear, compressing down to between 280K and 512K.


Cloning the sender's filesystem (nb-clone) is not available on XO-1.5. See [[Imaging_for_XO-1.5]] for an alternative.
* lzma is too slow for on-the-fly compression and would only be feasible for off-line preparation of a pre-compressed image. If network conditions are bad (low throughput) it might still be worthwhile.


=== Stopping ===
* gzip is marginal for on-the-fly compression, especially in conjunction with the FEC generation step (need to benchmark that on XO-1.5). It will be the rate-limiting step unless the network speed is rather slow. But it still might be the winner under typical conditions. The compression time might overlap almost perfectly with the network time on the sending side. Decompression is probably fast enough not to cause problems for the receiver.


You can stop the sender by typing the ESC key or by powering off. If you type any other key, the sender will pause and ask you if you want to stop. If you then type 'y', it will stop; other keys will resume the sending.
* lzop is the speed demon, but if the network is slow, the compression ratio penalty becomes significant. However, if gzip rate-limits transmission to say 1 MB/sec, while lzop would let it run at 2 MB/sec, then lzop would win handily (256 < 368).


=== "No quiet channels" - Forcing the Channel ===
It would be nice if we could take advantage of the fact that the data in the .zd file is already compressed, but I'm not sure how to fit that into the FEC scheme without a lot of surgery. I suppose we could coalesce adjacent compressed blocks of data until we get 128K, then unblock later. That could get fairly complicated...


The sender chooses a wireless channel automatically by scanning to find the least-busy channel. If they are all busy, i.e. they all have received signal strength values exceeding some threshold, the sender tell you which one has the least signal strength and ask for confirmation before proceeding on that channel.
=== Comments ===


You can force it to use a specific channel by appending either "1", "6", or "11" to one of the commands above, for example:
[dsd] ''I don't think the "cloning the senders disk" option is very useful though, and Iwas wondering if it would make your life easier if we forgot about that option.''


ok nb-update6
[wmb] Cloning was quite convenient in early testing of NB-1.0, as I didn't have to prep a stick when I made a protocol change - just download the changed firmware into a machine, turn it on, and voila it was a sender. In the 1.5 case, since internal and external SD cards are pretty much identical, cloning might be somewhat more useful in the deployment case than it was with XO-1. You could use it as a scalable mass-duplicator, for example, requiring no special setup. This brings up an interesting security issue: obviously cloning to the internal disk should be prevented when the client is secure, but what about cloning to an external SD card?


or
[dsd] ''Also we still have some flexibility in the .zd format, maybe we could make it more suitable for nandblasting.''


ok nb-secure11
[wmb] Excellent point. I suppose we could have a data format where the zdata is FEC-encoded and each sub-block separately compressed. That would make the .zd file somewhat larger, since the FEC redundancy overhead would be included (about 20% overhead) and the compression efficiency would be reduced (smaller blocks). So maybe 30% overhead overall as a guess. It wouldn't affect the non-NB fs-update time too badly, as the first N subblocks are actually the unencoded data, so no need to do FEC if you can read them reliably. It would allow you to recover from some read errors, if fs-update were able to do FEC decoding as necessary.

If you force the channel, the sender won't check if the channel is busy, it will just start sending.

=== Changing the Redundancy ===

You can change the redundancy percentage before you execute one of the send commands. For example, to set the sender redundancy to 12% (the default is 20%), type:

ok d# 12 to redundancy

The redundancy controls the number of extra error correction packets that the sender sends for each erase block worth of source data.

== Design Notes ==

[[Nandblaster_for_XO-1.5_Design_Discussion]]

Latest revision as of 00:57, 22 August 2012

Quick Start

XO-1.5

How to install to multiple XO-1.5 using NANDblaster:

The build is installed to the internal microSD device. You will need a USB drive of at least 1Gb capacity, but only for the duration of the installation.

  • Prepare the USB drive:
    • download the os*.zd file for the build you wish to install,
    • copy the file to the USB drive,
    • rename the file to fs.zd,
    • check that the USB drive contains the file fs.zd,
  • Prepare the sender laptop:
    • choose a laptop, it must be an XO-1.5, it need not have anything installed on it, and if there is a build installed on it won't be changed by the process,
    • check that the laptop has firmware q3a50 or later, upgrade the firmware if needed,
    • unlock the laptop if it is locked,
  • Start the sender laptop:
    • insert the USB drive in the laptop,
    • power up to the Ok prompt,
    • type nb-update and press enter,
    • wait for it to get going, it is ready when you see "pass 1" and a slowly increasing block count ... it usually takes about 15 seconds,
    • leave the USB drive in the laptop,
  • Start the receiver laptops, one by one or at will if you have many people helping:
    • hold down the four game keys while turning on each laptop, (you may here an interrupted startup sound, this is normal),
    • watch the display until the coloured boxes begin to appear,
  • If a receiver laptop does not properly start:
    • ensure they have firmware q3a45 or later, upgrade the firmware if needed,
    • try once more, wireless contention may have prevented association,
    • move it closer to the sender, try once more, radiofrequency noise may have prevented association,
  • Wait for all receiver laptops to finish. To assess the state of a laptop:
    • if the left-hand wireless LED is on, and a black outline is moving over the screen, then the laptop is listening for data,
    • if the blocks are mostly pink, noise is hindering reception,
    • if the blocks are mostly yellow, then noise. distance or a broken antenna is preventing reception; the laptop should be moved closer to the sender,
    • if the blocks are filling in green, and the storage LED is flickering, then the laptop is finished listening and is writing data to internal storage; the laptop may be moved away from the sender,
  • Stop the sender laptop:
    • press Esc.png Escape key on the sender laptop,
    • power it off,
    • remove the USB drive.

Starting the Sender

NANDblasting an Unsigned Filesystem Image File

To send an unsigned filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Insert the USB drive into the sending XO and type: 

 ok nb-update

NANDblasting a Signed Filesystem Image File

To send a signed filesystem image, put the filesystem image file (*.zd) on a USB drive as "/fs.zd". Put the corresponding signature bundle (the .zip that contains a control file and a signature file) on the USB drive as "/fs.zip". Insert the USB drive into the sending XO and type: 

 ok nb-secure

Cloning the sender's Filesystem

Cloning the sender's filesystem (nb-clone) is not available on XO-1.5. See Imaging_for_XO-1.5 for an alternative.

Stopping

You can stop the sender by typing the ESC key or by powering off. If you type any other key, the sender will pause and ask you if you want to stop. If you then type 'y', it will stop; other keys will resume the sending.

"No quiet channels" - Forcing the Channel

The sender chooses a wireless channel automatically by scanning to find the least-busy channel. If they are all busy, i.e. they all have received signal strength values exceeding some threshold, the sender tell you which one has the least signal strength and ask for confirmation before proceeding on that channel.

You can force it to use a specific channel by appending either "1", "6", or "11" to one of the commands above, for example: 

 ok nb-update6

or 

 ok nb-secure11

If you force the channel, the sender won't check if the channel is busy, it will just start sending.

Changing the Redundancy

You can change the redundancy percentage before you execute one of the send commands. For example, to set the sender redundancy to 12% (the default is 20%), type: 

 ok d# 12 to redundancy

The redundancy controls the number of extra error correction packets that the sender sends for each erase block worth of source data.

Design Notes

Nandblaster_for_XO-1.5_Design_Discussion

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=Nandblaster_for_XO-1.5&oldid=273683"