Testing checklist

Overview

We are a big believer in automated testing, to the extent that testing can be automated. Computers don't get bored, where as people do (and tend to skip tests). So making it as easy as possible to test our systems has a high priority.

Tinderbox is one such automation tool. In the OLPC tinderbox we can run multiple systems through many, but not all tests, while stressing the system. We reboot the systems, reinstall the image to be tested from scratch, run a set of tests (while loading the system), and repeat. You can see immediately if something has failed, and see where in the testing cycle the failure occurred. Some tinderboxes (e.g. mozilla's) are even able to identify the likely checkin that broke a build being tinderboxed, though that level of automation is currently out of our reach.

Equally important is that tinderboxes can automate performance tests, and arrange to systematically keep track of key performance and/or memory metrics over time, allowing quick identification of performance regressions.

Chris Ball heads OLPC's performance testing work: we're sure he can use help if you want to help in this area; there is always more that can be done! Not all the automatable tests have yet been implemented.

Automated(able) Tests

• Basic functionality
  • Seeing if the BIOS started
  • Running built in tests
• Networking:
  • Getting an IP address on ethernet
  • Copying an image from ethernet and checksumming it, before installing into NAND flash
  • Rebooting into NAND Flash
  • Rebooting into an image on disk
• Reinstallation of the base software image
• Wireless gets an IP address
  • Copying 10 meg from wireless, and checksumming it
• Running of CPU burn for a period
• Running a video player and dbench on NAND and SD (coming soon).
  • Getting an address on wireless
  • Copying 10 meg of data, and checksumming the result
• Installation and testing of the BIOS/OFW by
• Rebooting, and repeating the whole sequence

We need to acquire a programmable power switch, to allow cold reboot tests; right now, we can only perform warm reboots. So far, however, most problems have occurred during warm reboots.

Manual Tests

Not all tests can be automated, unfortunately,

For that tests you can see Board Test Plan

Software Tests

We are not yet set up with Sugar to automate any tests. The testing of user interface software is notoriously hard to automatically test and notoriously sensitive to the details of the programs being tested, though some testing can be done.