Nandblaster for XO-1: Difference between revisions
(New all-singing, all-dancing NANDblaster) |
|||
| Line 1: | Line 1: | ||
== OFW Multicast Wireless NAND Updater == |
== OFW Multicast Wireless NAND Updater (NANDblaster)== |
||
OFW Q2E22i [http://dev.laptop.org/~wmb/q222h.rom] (a test version) can update the NAND FLASH quickly via wireless. One XO is the sender. Any number of other XOs can receive simultaneously. The sender data comes either from a NAND image file (USB or SD) or from the sending machine's own NAND FLASH (cloning). The NAND image can be partitioned or not, signed or unsigned. The receiving machine can be secure or unsecure. |
|||
OFW Q2E19 [http://wiki.laptop.org/go/OLPC_Firmware_q2e19] has experimental support for receiving multicasted NAND FLASH updates via wireless. |
|||
== Starting the Sender == |
|||
It's in a fairly raw state at the moment (it needs performance, security, and user interface improvements), but it basically works. |
|||
The sending machine must be unsecure so you can type commands at the ok prompt. |
|||
To use it, say |
|||
=== Cloning the Sender's NAND === |
|||
ok wifi ''myessid'' |
|||
ok mcastnand |
|||
To clone the sender's own NAND FLASH, type: |
|||
The default value for ''myessid'' is OLPCOFW , so you could set your wireless AP to that SSID and omit the wifi command. That works with an open wireless network. You can supply authentication keys with one of: |
|||
ok nb-clone |
|||
ok wep HHHHHHHHHH \ 10 hex digits |
|||
ok wep HHH...HHH \ 26 hex digits |
|||
ok pmk HHH...HHHH \ 64 hex digits |
|||
When cloning, the sender displays a graphic progress report showing which of its NAND FLASH blocks is currently being sent. |
|||
PMK is the pairwise master key for WPA. |
|||
=== NANDblasting an Unsigned NAND Image File === |
|||
On the server, you do this: |
|||
To send an unsigned NAND image from a file, put the NAND image file on a USB key as "/fs.img". Put the "control file" (the file with partition: and eblock: lines) on the USB key as "fs.plc". Insert the USB key into the sending XO and type: |
|||
git clone git://git.infradead.org/mtd-utils.git |
|||
cd mtd-utils |
|||
make serve_image |
|||
./serve_image 239.255.1.2 12345 myimage.img 131072 100 |
|||
ok nb-update |
|||
Replace "myimage.img" with the name of a NAND image file. |
|||
The "100" parameter is the multicast data rate in Kbytes/sec. 100 is about the best you can do with garden-variety wireless access points, as |
|||
it corresponds to a bit rate of about 1 MBit/sec, which is the built-in multicast rate limit for a lot of access points. If you can configure your access point to multicast faster than that, you can bump the rate parameter up to 500. If you go any faster than that, OFW will start to lose packets and the overall performance will suffer (improving this is high on my list). |
|||
=== NANDblasting an Signed NAND Image File === |
|||
If you don't have a fast-multicasting access point, you can test the 500 rate with unicasting: (XXX this is currently broken in q2e19; it fails with "Can't open net" message. It's fixed in q2e19c and later.) |
|||
To send a signed NAND image from a file, put the NAND image file on a USB key as "/fs.img". Put the signature bundle (the .zip that contains a control file and a signature file) on the USB key as "fs.zip". Insert the USB key into the sending XO and type: |
|||
ok boot rom:mnandcast X.Y.Z.S,,X.Y.X.C 12345 /nandflash |
|||
ok nb-secure |
|||
X.Y.Z.S is the server's IP address, X.Y.X.C is the client's. |
|||
When sending from a file, the progress report is text. |
|||
./serve_image X.Y.Z.C 12345 myimage.img 131072 500 |
|||
== |
=== Options === |
||
You can stop the sender by typing the ESC key or by powering off. If you type any other key, the sender will pause and ask you if you want to stop. If you then type 'y', it will stop; other keys will resume the sending. |
|||
At a rate of 100, it will take about 10 seconds per megabyte, so 100 MBytes would take 1000 seconds/16 minutes, and 400 MBytes would take about an hour. That clearly sucks. The limitation at this level is the access point; the solutions are either to spec better access points or, preferably, work out how to use an XO as the sender (will require that OFW learns how to listen to the mesh). Sending to the mesh with TTL=1 is essentially the same as multicast. |
|||
The sender chooses a wireless channel automatically, by doing a scan to find the least-busy channel. If they are all busy, i.e. they all have received signal strength values exceeding some threshold, the sender will display an error message and fail. You can force the sender to use a specific channel by appending either "1", "6", or "11" to one of the commands above, for example: |
|||
At 500, it would take about 12 minutes. (Writing the FLASH takes a couple of minutes, so say 15 minutes for a typical OS image at 500). The limitation at the 500 level is OFW's USB Wlan driver, which does a lame job of bulk-in buffer management. With some work, I should be able to double that rate; 8 minutes for a 400 MByte image is my goal for the moment. (It turns out that improving the WLAN driver alone didn't help much; other bottlenecks were UDP checksum calculation and non-overlapped NAND FLASH access time. With the WLAN driver improved, the UDP checksummer optimized, and the NAND access stubbed out, I can receive at 1.2 MBytes/s, corresponding to a network time of 5 minutes for 400 MBytes. So if I can do a good job of FLASH access time overlap, the goal is in sight.). |
|||
ok nb-clone6 |
|||
Keep in mind that you can have a lot of XOs doing this at once, so you can start a few hundred and by the time you have finished starting them all, the first ones will probably be finished. So maybe the 10 to 15 minute time range is acceptable. |
|||
If you force the channel, the sender won't check if the channel is busy, it will just start sending. |
|||
== Security == |
|||
You can change the redundancy percentage before you execute one of the send commands. For example, to set the sender redundancy to 12% (the default is 20%), type: |
|||
Currently there is none. The plan to have the server also periodically send a signature file to validate the .img file. It will be basically the same set of files that we currently use for secure filesystem updates via the 4-button salute. |
|||
ok d# 12 to redundancy |
|||
== User interface == |
|||
The redundancy controls the number of extra error correction packets that the sender sends for each erase block worth of source data. |
|||
Could do something with the 4-button salute, but I'm also thinking of turning on the OFW GUI at some point. The 4-button approach has the advantage of being faster - turn on the power, hold the buttons, go to the next machine. |
|||
== Starting the Receiver == |
|||
The receiver can be secure or not. In either case, you can start reception by powering on then holding down the four game buttons above the power button (circle, square, check, X). That button combination runs the updater; the NANDblaster function is now one of the update choices. The updater first looks for fs.zip on USB, then on SD, then NANDblaster, and finally it tries to associate with an access point and update from an HTTP server. |
|||
If the receiver is not secure, you can start the NANDblaster by typing: |
|||
ok nb |
|||
The one reception command automatically handles all forms of sending - whatever the sender is sending, that's what the receiver will do. |
|||
The receiver automatically scans the three wireless channels (1, 6, 11) to find a sender. If a NANDblaster sender is transmitting on one of those channels, the receiver locks onto it and begins reception. You can force the receiver to use a specific channel by putting "1", "6", or "11" right after "nb", for example: |
|||
ok nb11 |
|||
Forcing the channel shouldn't be necessary in the usual case; if there is a sender, the receiver should find it. One situation where you might want to force the channel would be if you had multiple senders, perhaps sending different images. |
|||
== Performance == |
|||
The NANDblaster uses a lot of the wireless bandwidth, sending at about 1.4 MBytes/sec, depending somewhat on the source of the data. In a quiet RF environment, the receivers can accept that data rate with few errors. It's common for the receiver to acquire a complete set of data in one sending pass. If it doesn't get all the data on the first pass, it usually gets the rest on the second pass. |
|||
At that rate, one sending pass of a 250 MB image takes just over 3 minutes. After the receiver has gotten a complete set, it takes another couple of minutes to decode the error correction, check the hashes, and write the data to its NAND FLASH. |
|||
You can run as many receivers as you want at the same time, limited only by your ability to power them and place them within wireless range of the sender. The wireless traffic is strictly one way - sender to receiver. The receivers are passive - no beacons, no probes, no acknowledgments. The sender uses the mesh with TTL=1 so its packets aren't retransmitted by other mesh nodes that happen to be listening. Furthermore, the packets are at the Ethernet layer, with a special type code "XO", so devices that use standard protocols should ignore them. |
|||
== How it works == |
== How it works == |
||
The server sends data continuously. It divides each NAND eraseblock-sized chunk of the image into some number of packets (currently |
The server sends data continuously. It divides each NAND eraseblock-sized chunk of the image into some number of packets (currently 99), then it creates some more packets with redundant information using a Forward Error Correction scheme based on Vandermonde matrices. The redundancy percentage defaults to 20, so there are 119 packets for each eraseblock . The receiver can reconstruct the erase block contents from any set of 99 distinct packets. You can change the redundancy level from the ok prompt with: |
||
ok d# 10 to redundancy |
|||
ok nb-clone |
|||
In my tests in a very quiet RF environment, I've seen good results with redundancy as low as 3%, but I expect that a higher number is a better default. Increasing the redundancy increases the probability that the receiver will get a complete set in one pass, but it slows down each pass. |
|||
The receiver can start anywhere in the sequence. It collects packets, storing them on the NAND FLASH with a bit of overflow into RAM, until it has |
The receiver can start anywhere in the sequence. It collects packets, storing them on the NAND FLASH with a bit of overflow into RAM, until it has 99 distinct packets for each eraseblock. Then it does the mathematical magic to reconstruct the data, writing the reconstruction back to the NAND. The math essentially amounts to solving a set of simultaneous equations. |
||
If the packet error rate is reasonably low (less than about 10%), there's a good chance that the receiver will get a complete set of |
If the packet error rate is reasonably low (less than about 10%), there's a good chance that the receiver will get a complete set of 99 packets for every erase block in one server cycle. If not, the receiver keeps listening until it finally gets a complete set. I've tested this with error rates approaching 50%, and it still works, although it takes a long time. |
||
== Source Code == |
== Source Code == |
||
| Line 66: | Line 87: | ||
The source for the OFW version is currently at [http://dev.laptop.org/git?p=users/wmb/multicast-nand] |
The source for the OFW version is currently at [http://dev.laptop.org/git?p=users/wmb/multicast-nand] |
||
The |
The source for David Woodhouse's original Linux version is at git://git.infradead.org/mtd-utils.git . The OFW version has diverged a great deal from the original. The two no longer interoperate - I had to extend the protocol headers to handle partitions and security. |
||
== Background == |
== Background == |
||
Revision as of 08:49, 5 December 2008
OFW Multicast Wireless NAND Updater (NANDblaster)
OFW Q2E22i [1] (a test version) can update the NAND FLASH quickly via wireless. One XO is the sender. Any number of other XOs can receive simultaneously. The sender data comes either from a NAND image file (USB or SD) or from the sending machine's own NAND FLASH (cloning). The NAND image can be partitioned or not, signed or unsigned. The receiving machine can be secure or unsecure.
Starting the Sender
The sending machine must be unsecure so you can type commands at the ok prompt.
Cloning the Sender's NAND
To clone the sender's own NAND FLASH, type:
ok nb-clone
When cloning, the sender displays a graphic progress report showing which of its NAND FLASH blocks is currently being sent.
NANDblasting an Unsigned NAND Image File
To send an unsigned NAND image from a file, put the NAND image file on a USB key as "/fs.img". Put the "control file" (the file with partition: and eblock: lines) on the USB key as "fs.plc". Insert the USB key into the sending XO and type:
ok nb-update
NANDblasting an Signed NAND Image File
To send a signed NAND image from a file, put the NAND image file on a USB key as "/fs.img". Put the signature bundle (the .zip that contains a control file and a signature file) on the USB key as "fs.zip". Insert the USB key into the sending XO and type:
ok nb-secure
When sending from a file, the progress report is text.
Options
You can stop the sender by typing the ESC key or by powering off. If you type any other key, the sender will pause and ask you if you want to stop. If you then type 'y', it will stop; other keys will resume the sending.
The sender chooses a wireless channel automatically, by doing a scan to find the least-busy channel. If they are all busy, i.e. they all have received signal strength values exceeding some threshold, the sender will display an error message and fail. You can force the sender to use a specific channel by appending either "1", "6", or "11" to one of the commands above, for example:
ok nb-clone6
If you force the channel, the sender won't check if the channel is busy, it will just start sending.
You can change the redundancy percentage before you execute one of the send commands. For example, to set the sender redundancy to 12% (the default is 20%), type:
ok d# 12 to redundancy
The redundancy controls the number of extra error correction packets that the sender sends for each erase block worth of source data.
Starting the Receiver
The receiver can be secure or not. In either case, you can start reception by powering on then holding down the four game buttons above the power button (circle, square, check, X). That button combination runs the updater; the NANDblaster function is now one of the update choices. The updater first looks for fs.zip on USB, then on SD, then NANDblaster, and finally it tries to associate with an access point and update from an HTTP server.
If the receiver is not secure, you can start the NANDblaster by typing:
ok nb
The one reception command automatically handles all forms of sending - whatever the sender is sending, that's what the receiver will do.
The receiver automatically scans the three wireless channels (1, 6, 11) to find a sender. If a NANDblaster sender is transmitting on one of those channels, the receiver locks onto it and begins reception. You can force the receiver to use a specific channel by putting "1", "6", or "11" right after "nb", for example:
ok nb11
Forcing the channel shouldn't be necessary in the usual case; if there is a sender, the receiver should find it. One situation where you might want to force the channel would be if you had multiple senders, perhaps sending different images.
Performance
The NANDblaster uses a lot of the wireless bandwidth, sending at about 1.4 MBytes/sec, depending somewhat on the source of the data. In a quiet RF environment, the receivers can accept that data rate with few errors. It's common for the receiver to acquire a complete set of data in one sending pass. If it doesn't get all the data on the first pass, it usually gets the rest on the second pass.
At that rate, one sending pass of a 250 MB image takes just over 3 minutes. After the receiver has gotten a complete set, it takes another couple of minutes to decode the error correction, check the hashes, and write the data to its NAND FLASH.
You can run as many receivers as you want at the same time, limited only by your ability to power them and place them within wireless range of the sender. The wireless traffic is strictly one way - sender to receiver. The receivers are passive - no beacons, no probes, no acknowledgments. The sender uses the mesh with TTL=1 so its packets aren't retransmitted by other mesh nodes that happen to be listening. Furthermore, the packets are at the Ethernet layer, with a special type code "XO", so devices that use standard protocols should ignore them.
How it works
The server sends data continuously. It divides each NAND eraseblock-sized chunk of the image into some number of packets (currently 99), then it creates some more packets with redundant information using a Forward Error Correction scheme based on Vandermonde matrices. The redundancy percentage defaults to 20, so there are 119 packets for each eraseblock . The receiver can reconstruct the erase block contents from any set of 99 distinct packets. You can change the redundancy level from the ok prompt with:
ok d# 10 to redundancy ok nb-clone
In my tests in a very quiet RF environment, I've seen good results with redundancy as low as 3%, but I expect that a higher number is a better default. Increasing the redundancy increases the probability that the receiver will get a complete set in one pass, but it slows down each pass.
The receiver can start anywhere in the sequence. It collects packets, storing them on the NAND FLASH with a bit of overflow into RAM, until it has 99 distinct packets for each eraseblock. Then it does the mathematical magic to reconstruct the data, writing the reconstruction back to the NAND. The math essentially amounts to solving a set of simultaneous equations.
If the packet error rate is reasonably low (less than about 10%), there's a good chance that the receiver will get a complete set of 99 packets for every erase block in one server cycle. If not, the receiver keeps listening until it finally gets a complete set. I've tested this with error rates approaching 50%, and it still works, although it takes a long time.
Source Code
The source for the OFW version is currently at [2]
The source for David Woodhouse's original Linux version is at git://git.infradead.org/mtd-utils.git . The OFW version has diverged a great deal from the original. The two no longer interoperate - I had to extend the protocol headers to handle partitions and security.
Background
David Woodhouse developed this scheme while at Quanta and also tested it in Mongolia. Mitch Bradley ported it to OFW and is the current owner.
The following captures an IRC discussion that contains important information about a multicast-distribution scheme for updating XOs wirelessly.
<erikg> Mitch_Bradley: XS-based wireless reflash is a *good* idea. i read some emails on the topic, but am not entirely clear on the issues which are involved. <Mitch_Bradley> erikg: XS-based reflash should work in principle. I tested reflashing over wireless on my home network with success. <Mitch_Bradley> erikg: the issue is whether the XS network is discoverable via the essids mentioned <Mitch_Bradley> erikg: and also there were some problems with OFWs support of mesh-mode for the wlan <erikg> Mitch_Bradley: turning on a bunch of laptops and letting them sit takes considerably less manual effort than the usb-based (serial) approach. in peru it would help them to reduce the warehouse staff, most of whom do double-duty activating and physically distributing the laptops. <Mitch_Bradley> erikg: OFW works well in access point mode, but how well it works in ad hoc mode is questionable <erikg> Mitch_Bradley: in OFW wireless reflash the reflash is the same aside from the source of the data? <erikg> Mitch_Bradley: i'm only suggesting use in AP mode. <Mitch_Bradley> erikg: yes <erikg> Mitch_Bradley: and can it use multicast? <erikg> Mitch_Bradley: e.g. so there can be a bunch of laptops turned on which then get the data? <Mitch_Bradley> erikg: no, OFW doesn't have support for multicast yet * erikg thought he saw an email from david woodhouse on the topic, but maybe his memory fails him <Mitch_Bradley> erikg: dwmw2 developed a multicast update scheme that includes forward error correction <erikg> Mitch_Bradley: forward error correction? <Mitch_Bradley> erikg: the plan was for OFW to include support for that protocol at some point <Mitch_Bradley> erikg: if you do straight multicast , the probability of an individual client missing a packet is rather high <erikg> Mitch_Bradley: right. the emails i read mentioned this. <Mitch_Bradley> erikg: so each client probably has to wait quite a few overall transmits before it has a complete set <Mitch_Bradley> erikg: if you send each packet with error correction codes, you can greatly reduce the probability of botched packets <erikg> Mitch_Bradley: i see. so the ECC work and multicast work need to be done? or has dwmw2 already completed the multicast work? <Mitch_Bradley> erikg: dwmw2 has a working multicast scheme that requires booting a small ram-based Linux on the clients <erikg> Mitch_Bradley: interesting. perhaps that would provide much more flexibility <Mitch_Bradley> erikg: so, while it would be nice to have it directly in OFW, that's not strictly necessary <Mitch_Bradley> erikg: OFW would be advantageous to reduce the network congestion from several laptops all trying to boot that ram-based Linux <dwmw2> erikg: http://gallery.infradead.org/main.php?g2_itemId=1540 <erikg> Mitch_Bradley: i guess it depends on how big the ram-based linux is <dwmw2> that's the machines being installed in Mongolia, all at once over wireless+multicast. <dwmw2> http://david.woodhou.se/olpc-nandcast.tar.gz <dwmw2> or something like that <dwmw2> you don't need much in it <dwmw2> just the recv_image program from the mtd-utils, basically. And a script which brings the network up and runs it. <erikg> dwmw2: what's needed on the server side? <dwmw2> send_image :) <dwmw2> sorry, serve_image <Mitch_Bradley> dwmw2: wasn't there some problem with some APs? <Mitch_Bradley> like, multicast being throttled to a ridiculously low data rate? <dwmw2> some APs only send multicast at the lowest possible rate (1Mb/s). Which sucks. When I first set it up I got an AP which lets you set the multicast rate (which was left in smithbone's custody and is around 1cc somewhere iirc). <dwmw2> more recently (and in Mongolia) I was just doing it over mesh not infra mode, with a libertas dongle in my shinybook <dwmw2> remember to set the mesh ttl to 1 and the mesh broadcast rate to something sane (like 11Mb/s iirc) <erikg> dwmw2: because if you set it too high the error rate goes way up? <dwmw2> becaue if you set it too low, you die of boredom <erikg> dwmw2: haha <erikg> dwmw2: would you please email me information about the brand of AP in question? <erikg> brand/version <dwmw2> I think it was a Buffalo one <erikg> ok <erikg> they've god 40k machines to flash here in peru and they're doing it by usb dongle <erikg> dwmw2: so what happens on the xo-side? <dwmw2> I went round the mall by the hotel in Shanghai, took photos of all the APs I could see, went back to the hotel and googled for their manuals. <dwmw2> then went back and bought the one which let you configure the multicast rate <erikg> hahaa <dwmw2> sending from a libertas dongle is probably easier <erikg> oh i mean via a usb flash memory <erikg> dwmw2: what do you have to do on the xo-side? what's the procedure? <dwmw2> for the XO side see that tarball. <dwmw2> boot into an initrd with a script which brings the network up and runs the rx tool <dwmw2> you only have to plug the usb key in for long enough for the kernel to start booting, then you can rip it out and move on to booting the next machine <erikg> so you have to flash this onto all the machines <erikg> right <dwmw2> ∀ machine, insert key, power on, wait a few seconds, remove key <erikg> dwmw2: this appears to be lacking the server code <erikg> or am i misunderstanding something? <dwmw2> that's for the XO side. <erikg> right <dwmw2> the server code is in the mtd-utils <dwmw2> git.infradead.org/mtd-utils.git <erikg> serve_images <erikg> i've git it before <erikg> serve_image <dwmw2> ./serve_image ff0f::1234 1234 myimage.img 131072 1800 <dwmw2> or something like that <erikg> cool! <dwmw2> 239.255.255.1 or something if you want Legacy IP instead of IPv6 (but then you have to set up Legacy IP addresses too, on the client side) <erikg> do you know what other deployments are using this? <dwmw2> I think I did that at one point -- my script in the tarball I showed you probably sets a 10.x.x.x address using the last three bytes of the MAC address? <erikg> or is it just mongolia <dwmw2> it was just Mongolia, while I was there. <erikg> i am seeing olpc.fth, vmlinuz, and initrd.img ... the script is in the initrd <erikg> are they still using this method? <Mitch_Bradley> they don't have dwmw2's shinybook <dwmw2> they don't need it -- just a libertas dongle. But we never really polished it enough for them to be able to use it. <erikg> hmmmm <dwmw2> We _could_ start streaming the image from the servers we have in place, but we haven't <dwmw2> the serve_image program just spews the image out (with 100% FEC) over and over again. <dwmw2> it's the only thing that needs to send on the network. <dwmw2> makes the network not very usable, when it's running full-stream. Even kills my Bluetooth mouse :) <dwmw2> but installs a whole lot of laptops nice and fast. <dwmw2> the hardest part was plugging all the damn things in for long enough to do the OF upgrade :) <dwmw2> the other reason it's not an 'official' install method is because the tool doesn't check the signature of the images <dwmw2> it wouldn't be hard to handle that (I think we talked about not writing the first block of the image until it was all checked and the signature passed; writing some marker there which told OF not to use it?) dwmw2> but we never did it. So we had a signed key with my XO-side stuff on it, which had a list of the serial numbers for Mongolia and would _only_ work on those laptops. <dwmw2> but would let you install _anything_ on those laptops, of course. <erikg> dwmw2: i see <dwmw2> if we want to be able to use multicast 'in anger', we should fix that -- it shouldn't be hard. <dwmw2> I think writing a new node type which is unknown to any existing JFFS2 implementation and has the 'cannot mount if unknown' compatibility bitmask ought to be sufficient -- write that in the first block of the flash when we start to rx the image. <dwmw2> then only remove it when the image is completely received and verified. Mitch_Bradley> The OFW updater has support for such a feature. It uses a marker in the otherwise-unused portion of the OOB area <dwmw2> we could use that then, since marking it unusable in OFW should be perfectly sufficient.