Collaboration network testbed

While the OLPC Mesh Testbed is conveniently located in the same space used by OLPC for development and business activities, it suffers from an excess of unrelated wireless traffic. In order to allow repeatable testing of larger numbers of laptops, a larger testbed has been set up at another location in the Boston area, where the 802.11b spectrum is relatively unused.

Test Setup

At this location, we currently have a hundred laptops and a school server, along with a separate packet recording machine. Ten laptops are grouped together on each 1.5 m by 0.7 m table. We hope to grow to at least a hundred and fifty laptops over time.

While we will eventually provide remote automated control of large portions of the testing, we are currently performing tests manually.

The laptop developer keying, upgrading, and application installation is done using USB keys.

Tests and Results

The tests are grouped according to the type of network used when they were performed: simple mesh, a mesh with a school server, or access points with a school server.

All packet traces are in wireshark format, and contain beacon and link layer management frames as well as IP packets. They were obtained using tcpdump -i rtapX -w filename.

More specifically, interfaces are configured to the appropriate channels at boot time via ifcfg_ethX files in /etc/sysconfig/network-scripts/ (eth1 to channel 1, eth2 to channel 11, and eth3 to channel 6). There is a script, setup_snoop, for setting up the interfaces which is run once after boot. Another, do_snoop, is used to start logging. Logging is stopped with sudo killall tcpdump

Simple Mesh

In these tests, the laptops are connected solely to one another, through the wireless mesh.

Startup

Startup is also frequently included in other tests, such as test 0321A and test 0321C.

Test 0317A

Laptop Build: 699

This was a staged startup of laptops. The packet trace was started five minutes before starting to turn the laptops on. The laptops were turned on in groups of five, with each group allowed to enter simple mesh mode before starting the next group.

The first ten laptops (X50 - X59) came up without a problem, with all laptops able to see the others.

One laptop (X69) in the third group of laptops (X65 - X69) came up without joining a simple mesh (Trac 5848). A reboot fixed the problem.

Another laptop (X73) in the fifth group of laptops (X70 - X74) came up with Trac 5848 and had to be rebooted before joining the simple mesh.

After rebooting two laptops, all 25 laptops (X50 - X74) could see the other 24 in their neighborhood view.

Packet Traces: Chan 1, Chan 11, Chan 6

Discussion of packet traces

Write Sharing

Test 0321B

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Write 55

This test started with 10 laptops (X50 - X59) in a simple mesh on channel 1 (test 0321A had just completed).

The Write activity was started on laptop X54, and shared with the neighborhood. The shared activity was seen on all laptops.

Nine laptops joined the shared activity, almost simultaneously. All of the laptops correctly launched the application and mirrored the shared document. All laptops were able to type and have their characters appear on the other laptops. It was very slow, especially if any characters were erased on any laptop. But it worked!

Packet Traces: Chan 1, Chan 11, Chan 6

Test 0317B

Laptop Build: 699 + Write 55

This test started with 25 laptops (X50 - X74) in a simple mesh on channel 1.

The Write activity was started on laptop X69 (169.254.1.204), and shared with the neighborhood.

The shared activity was seen on 24 of 25 laptops (X70 not showing). After further shared activity, it was eventually seen by all 25 laptops.

Ten laptops joined the shared activity, almost simultaneously. Three of the ten laptops launched write, but never showed the contents of the write document. The other seven correctly mirrored the shared document. The three laptops which failed to collaborate were X51 (169.254.5.153), X53 (169.254.2.7), and X54 (169.254.5.25).

More words were typed on X69, and were mirrored to the seven working laptops (no change in X51, X53, or X54).

Some words were typed on X59, and were correctly mirrored to the seven other collaborating laptops.

Packet Traces: Chan 1, Chan 11, Chan 6

Read Sharing

Test 0321A

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Read 44

This test booted up 10 laptops (X50 - X59). These laptops had been registered with a school server which was not present during the test. All laptops successfully joined the simple mesh on channel 1. All laptops saw one another in their neighborhood view.

The Read activity was started on laptop X59, using a small document (40 KB). The activity was then shared with the neighborhood. The shared activity was seen on all laptops.

An attempt to join the shared Read activity from one laptop (X58) failed. The activity launched, but the document was never displayed.

A later attempt to join the shared Read activity from another laptop (X57) failed. The activity launched, but the document was never displayed.

Packet Traces: Chan 1, Chan 11, Chan 6

Chat Sharing

Test 0321C

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Chat 35

This test booted up 21 laptops (X50 - X59, X61 - X72). These laptops had been registered with a school server which was not present during the test. Twenty of twenty-one laptops successfully joined the simple mesh on channel 1.

The failing laptop (X65) had no network circle in the home view (Trac 5848), and was rebooted after its logs were captured. It successfully joined the simple mesh. All laptops saw one another in their neighborhood view.

The Chat activity was started on laptop X59. The activity was then shared with the neighborhood (why does Chat have to be shared ? Is OLPC encouraging kids to Chat with themselves ?). Realized that this was done before laptop had joined simple mesh. Stopped activity.

Started Chat activity on laptop X59 again. The activity was then shared with the neighborhood. The shared activity was seen on all laptops.

An attempt to join the shared Chat activity from three laptops (X52, X53, X54) worked. A subsequent attempt to join with the remaining 17 laptops succeeded on all laptops.

All laptops were able to contribute to the conversation, and all laptops saw all characters typed on any laptop. Yahoo!

Packet Traces: Chan 1, Chan 11, Chan 6

Laptop X65 logs: All, dmesg, /var/log/messages, .sugar/default/nm/, Telepathy/Sugar

School Server Mesh

In these tests, a school server equipped with one to three active antennas provides the interconnection for the laptops. In general, the laptops are registered with the school server and using the school's presence service.

Startup & Registration

Startup is also frequently included in other tests, such as Chat (test 0323E), or Presence (test 0321D and test 0317E).

Test 0323A

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a simultaneous startup of 10 laptops, located on a single 0.7m x 1.5m table. 4 out of the 10 had problems finding the school mesh portal.

Ten laptops (X50 - X59) were turned on sequentially, with around one second between each startup. Each laptop had previously been registered with the school server. Four of the ten laptops (X51, X52, X57, X59) did not find the school server mesh portal, and started a simple mesh instead.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Laptop X59 logs: All, dmesg, /var/log/messages, Telepathy/Sugar

Server logs: /var/log/messages, laptop list

Test 0323B

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a simultaneous startup of 10 laptops, which were spatially dispersed around a 2m x 3m space. 2 out of the 10 had problems finding the school mesh portal.

Ten laptops (X51, X52, X57, X59, X69, X63, X71, X73, X76, X79) were turned on sequentially, with around one second between each startup. Each laptop had previously been registered with the school server. Two of the ten laptops (X52, X76) did not find the school server mesh portal, and started a simple mesh instead.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Laptop X52 logs: All, dmesg, /var/log/messages, Telepathy/Sugar

Server logs: /var/log/messages, laptop list

Test 0323C

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a simultaneous startup of 15 laptops, which were spatially evenly dispersed around a 2m x 3m space, with no laptops closer than 30 cm. to another unit in the test. 4 out of the 15 had problems finding the school mesh portal.

Fifteen laptops (X50, X52, X54, X56, X58, X62, X63, X65, X67, X69, X70, X72, X74, X77, X79) were turned on sequentially, with around one second between each startup. Each laptop had previously been registered with the school server. Four of the ten laptops (X54, X67, X69, X77) did not find the school server mesh portal, and started a simple mesh instead.

After the test was started, it was noticed that X57 (adjacent to X54 and X67) had been left sleeping instead of powered off, invalidating the even dispersion assumption of this test.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Server logs: /var/log/messages, laptop list

Test 0323D

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a simultaneous startup of 10 laptops, which were spatially evenly dispersed around a 5m x 5m space, with no laptops closer than 60 cm. to another unit in the test. 4 out of the 10 had problems finding the school mesh portal.

Ten laptops (X52, X55, X59, X62, X64, X65, X69, X70, X76, X78) were turned on sequentially, with around two seconds between each startup. Each laptop had previously been registered with the school server. Four of the ten laptops (X65, X69, X70, X78) did not find the school server mesh portal, and started a simple mesh instead.

For this test, the table containing X60 - X69 was moved outside the room where it is shown in the test diagram. The school server antenna was located in the center of the space, about 1.5 m over the plane containing the laptops.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Laptop X70 logs: All, dmesg, /var/log/messages, Telepathy/Sugar

Server logs: /var/log/messages, laptop list

Test 0323H

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a staggered startup of 20 laptops. 2 out of the 20 had problems finding the school mesh portal.

Twenty laptops (X50 - X59, X70 - X79) were turned on sequentially, two at a time with around 15 seconds between each pair. All laptops had previously been registered with the school server. Two of the twenty laptops (X70, X58) did not find the school server mesh portal, and started a simple mesh instead.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Server logs: /var/log/messages, laptop list

Test 0317D

Laptop Build: 699

School Server Build: 160

School Server Channels: 1

This test was a staged registration of 29 laptops. 7 out of the 29 had problems finding the school mesh portal. Once they found the portal, all of the laptops registered on the first try.

Ten laptops (X50 - X59) were turned on first. They all found the school server mesh portal. They were registered, the registration verified (on the Home Screen) and Sugar restarted using CTL-ALT-ERASE.

Another ten laptops (X61 - X70) were turned on. Four (X62, X64, X65, and X68) didn't find the school mesh portal, and joined a simple mesh on channel 1 (Trac 5963). Clicking on mesh channel 1 in the neighborhood view caused three of the laptops to find the school mesh portal, and trying twice worked on the fourth (X65).

Another nine laptops (X71 - X79) were turned on. Three of these didn't find the school mesh portal (Trac 5963), but all three found it on the first manual retry.

All of the laptop registered on the first try, but registration wasn't simultaneous.

Packet Traces: Chan 1, Chan 11, Chan 6

Test 0317C

Laptop Build: 699

School Server Build: 160

School Server Channels: 1

This test was supposed to be a staged registration of laptops. It was aborted after three (X52 - X54) of the first five laptops (X50 - X54) to be turned on couldn't find a school mesh portal and instead joined a simple mesh.

The school server active antenna had been placed on the same plane as the laptops (at 1 m from the floor). The antenna was moved to about 2 m off the floor, and the test repeated as 0317D.

Packet Traces: Chan 1, Chan 11, Chan 6

Presence

Test 0321D

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a simultaneous booting up of 29 registered laptops and 9 unregistered ones. 15 out of the 38 had problems finding the school mesh portal (Trac 4153, 5963). Most found the mesh portal after the mesh 1 circle in the neighborhood view was clicked on, but three (X71, X92, X96) required a reboot. Once they found the portal, all of the unregistered laptops registered on the first try and all connected using gabble.

29 of the laptops (X50 - X59, X61 - X79) had previously been registered with the school server (and the school ejabberd server). They were upgraded to have telepathy-salut-0.2.3-1.olpc2.i386.rpm and all logs (libertas, telepathy, sugar) turned on. Another nine unregistered laptops (X90 - X98) were used without the upgraded telepathy-salut. Salut shouldn't be used when connected to a school server, so no difference was expected. They were all turned on within a ninety second period, starting with X50 and moving up.

23 of the 38 laptops found the school server mesh portal. The ones that didn't (Trac 4153, 5963) were X53, X54, X58, X59, X64, X71, X73, X74, X76, X77, X79, X91, X92, X95, and X96. Note less correlation with startup time than seen in Test 0317E.

Twelve of the failing laptops found the school mesh portal after clicking on mesh channel 1 in the neighborhood view. Three of the failing laptops (X71, X92, and X96) didn't find it after two clicks, but did find it immediately upon reboot.

Two of the laptops (X91 aka 00:17:C4:11:14:5D aka 172.18.10.236, and X92 aka 00:17:C4:0D:14:54 aka 172.18.11.234) entered a weird state after having Sugar rebooted to force an ejabberd registration after a registration of the laptop with the school server. Unfortunately, no logs were obtained and the problem disappeared on reboot. More details later...

Packet Traces: Chan 1, Chan 11, Chan 6

Laptop X59 logs: All, dmesg, /var/log/messages, Telepathy/Sugar

Laptop X71 logs: All, dmesg, /var/log/messages, Telepathy/Sugar

Server logs: /var/log/messages, output of school_who, laptop list

Discussion of Packet Traces

Test 0317E

Laptop Build: 699

School Server Build: 160

School Server Channels: 1

This test was a simultaneous booting up of 29 registered laptops. 8 out of the 29 had problems finding the school mesh portal (Trac 5963). Once they found the portal, all of the laptops connected using gabble (although three had to have sugar restarted before they would display neighbors properly.)

All 29 laptops (X50 - X59, X61 - X79) had previously been registered with the school server (and the school ejabberd server). They were turned on within a sixty second period, starting with X50 and moving up.

21 of the 29 laptops found the school server mesh portal. The ones that didn't (Trac 5963) were X57, X65, X68, X70, X72, X75, X78, and X79. Note that these were the last ones turned on!

All 8 failing laptops found the school mesh portal after clicking on mesh channel 1 in the neighborhood view. Three of the failing laptops (X65, X68, and X75) didn't show other laptops in their neighborhood view until Sugar was restarted. They were not running salut instead of gabble (they didn't show each other), instead they didn't show any other XOs at all.

Packet Traces: Chan 1, Chan 11, Chan 6

Server logs: /var/log/messages, later output of school_who

Discussion of Packet Traces

Chat

Test 0323E

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Chat 35

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a staggered startup of 15 laptops, which were spatially evenly dispersed around a 5m x 5m space, with no laptops closer than 20 cm. to another unit in the test. 2 out of the 15 had problems finding the school mesh portal. Then another 22 laptops were turned on slowly, and a chat session between nine of the thirty-seven laptops was tested. It worked fine.

Fifteen laptops (X50, X52, X54, X56, X58, X62, X63, X64, X65, X69, X70, X72, X74, X77, X79) were turned on sequentially, with around ten seconds between each startup. Each laptop had previously been registered with the school server. Two of the ten laptops (X56 and X58) did not find the school server mesh portal, and started a simple mesh instead.

The two failing laptops immediately joined the mesh when mesh channel 1 was clicked on in the neighborhood view. It took up to two minutes for the neighborhood view to reflect the newfound connection to a school server.

After a minute, another twenty-two laptops (others from X50 - X79 and X90 - X98) were turned on sequentially, with around ten seconds between each startup. Another four or five laptops failed to find the school mesh portal, but found it after restarting the network association process by clicking on mesh channel 1 in the neighborhood view.

Chat was started on laptop X65, and it was shared with the neighborhood.

In 60 sec, 31 laptops were showing the shared activity on their neighborhood view. The ones which didn't included X59, X63, X93, and X96). (Trac ticket ?, logs attached)

The Chat activity was joined sequentially, with around two seconds between join, by laptops X62, X64, X66 - X69, X54, and X74. All laptops successfully joined, and were able to type and have it seen by all other participants.

For this test, the table containing X60 - X69 was moved outside the room where it is shown in the test diagram. The school server antenna was located in the center of the space, about 1.5 m over the plane containing the laptops. The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Laptop X59 logs: All, dmesg, /var/log/messages, Telepathy/Sugar

Server logs: /var/log/messages, laptop list

Write Sharing

Test 0321E

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Write 55

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test started with 37 students connected to a school server (subsequent to test 0321D). All were registered with the server and using Telepathy-Gabble. All students could see one another in their neighborhood views.

Started Write activity on X69 aka 00:17:c4:0d:00:01 aka 172.18.10.248. Shared the activity with the neighborhood. It was seen by all students.

The activity was successfully joined by five students (X70 - X74).

Started another Write activity on X58. Shared the activity with the neighborhood. It was seen by all students.

This second activity was successfully joined by five students (X61 - X65).

Characters were typed by X69 through X74, they were all seen by all activity participants.

Typed on X61 and crashed write. The first character typed was successfully sent to all laptops. I was frustratingly unable to get the logs as the keyboard had a stuck CTL key !!!

Packet Traces: Chan 1, Chan 11, Chan 6

Read Sharing

Test 0323F

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Read 44

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a simultaneous startup of 10 laptops, which were located on a 0.7m x 1.5m table. 2 out of the 10 had problems finding the school mesh portal. All were successfully registered, then a read activity was successfully shared to all laptops.

Ten laptops (X20 - X29) were turned on sequentially, with around one seconds between each startup. None of the laptops had previously been registered with the school server. Two of the ten laptops (X27 and X28) did not find the school server mesh portal, and started a simple mesh instead.

X20 - X26 and X29 were registered with the school server (all within a second), and Sugar was restarted.

X27 and X28 associated immediately with the school mesh portal when mesh channel 1 was clicked on. They too registered immediately and had Sugar restarted.

All laptops saw all other laptops in their neighborhood view.

A PDF document (VGA Pinout, about 40 KB), stored on laptop X25, was opened using Read. The activity was shared with the neighborhood. The shared activity was seen by all laptops.

The shared Read activity was joined by X26, and worked! X27, X28, and X29 then joined within a couple of seconds of each other. They too were successful (although it took around a minute for X29 to load the document). Finally, laptops X20 - X24 joined. All successfully loaded the shared document.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Server logs: /var/log/messages, laptop list

Test 0323G

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm + Read 44

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: 1

This test was a staggered startup of 10 laptops, which were located on a 0.7m x 1.5m table. 1 out of the 10 had problems finding the school mesh portal. A read activity was successfully shared to all laptops.

Ten laptops (X20 - X29) were turned on sequentially, with around five seconds between each startup. All of the laptops had previously been registered with the school server. One of the ten laptops (X28) did not find the school server mesh portal, and started a simple mesh instead.

X28 associated immediately with the school mesh portal when mesh channel 1 was clicked on.

All laptops saw all other laptops in their neighborhood view.

A PDF document (VGA Pinout, about 40 KB), stored on laptop X24, was opened using Read. The activity was shared with the neighborhood. The shared activity was seen by all laptops.

The shared Read activity was joined by laptops X20 - X23 and X25 - X29, with two or three seconds between joins. All successfully loaded the shared document.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Server logs: /var/log/messages, laptop list

School Server WiFi

In these tests, the laptops are connected to the school server through normal 802.11b/g access points.

Presence

Test 0323J

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: Linksys WiFi AP on channel 1

This test was a simultaneous startup of 37 laptops. It was determined that the AP only supported up to thirty simultaneous connections. Other laptops were unable to connect to the school server.

Thirty-seven laptops (X50 - X59, X62 - X79, X90 - X98) were turned on sequentially, with around one sec between each one. All laptops had previously been registered with the school server. All laptops started a simple mesh. Manually, they each were instructed to use the linksys access point by clicking on its circle (icon) in the neighborhood view.

The first thirty laptops connected to the school server, and displayed presence information about all the other connected laptops. The remaining seven laptops (X54, X59, X64, X75, X90, X91, and X98) were unable to associate with the AP.

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Server logs: /var/log/messages, laptop list

Test 0323K

Laptop Build: 699 + telepathy-salut-0.2.3-1.olpc2.i386.rpm

School Server Build: 160 + 22.p6 Libertas firmware

School Server Channels: Linksys WiFi AP on channel 1

This test was a simultaneous startup of 30 laptops, previously connected to a school server through an AP. None of the laptops tried to connect with the previous AP. 29 of them formed a simple mesh, with one deciding not to talk to anybody.

Thirty laptops (X50 - X53, X55 - X58, X62 - X63, X65 - X74, X76 - X79, X92 - X97) were turned on sequentially, with around one sec between each one. All laptops had previously been registered with the school server, and had used this AP the last time they were powered on.

All laptops started a simple mesh, except for X55, which didn't show any network connection (Trac 5848).

I was surprised. I thought the default behavior was to associate with the previous network, if found!

The logging was conducted with three Active Antennas, separated by one meter, all on channel 1.

Packet Traces: Chan 1 (a), Chan 1 (b), Chan 1 (c)

Server logs: /var/log/messages, laptop list