Testing checklist

Overview

We are a big believer in automated testing, to the extent that testing can be automated. Computers don't get bored, where as people do (and tend to skip tests). So making it as easy as possible to test our systems has a high priority.

Tinderbox is one such automation tool. In the OLPC tinderbox we can run multiple systems through many, but not all tests, while stressing the system. We reboot the systems, reinstall the image to be tested from scratch, run a set of tests (while loading the system), and repeat. You can see immediately if something has failed, and see where in the testing cycle the failure occurred. Some tinderboxes (e.g. mozilla's) are even able to identify the likely checkin that broke a build being tinderboxed, though that level of automation is currently out of our reach.

Equally important is that tinderboxes can automate performance tests, and arrange to systematically keep track of key performance and/or memory metrics over time, allowing quick identification of performance regressions.

Chris Ball heads OLPC's performance testing work: we're sure he can use help if you want to help in this area; there is always more that can be done! Not all the automatable tests have yet been implemented.

Automated(able) Tests

• Basic functionality
  • Seeing if the BIOS started
  • Running built in tests
• Networking:
  • Getting an IP address on ethernet
  • Copying an image from ethernet and checksumming it, before installing into NAND flash
  • Rebooting into NAND Flash
  • Rebooting into an image on disk
• Reinstallation of the base software image
• Wireless gets an IP address
  • Copying 10 meg from wireless, and checksumming it
• Running of CPU burn for a period
• Running a video player and dbench on NAND and SD (coming soon).
  • Getting an address on wireless
  • copying 10 meg of data, and checksumming the result
• Installation and testing of the BIOS/OFW by
• Rebooting, and repeating the whole sequence

We need to acquire a programmable power switch, to allow cold reboot tests; right now, we can only perform warm reboots. So far, however, most problems have occurred during warm reboots.

Manual Tests

Not all tests can be automated, unfortunately. The testing of user interface software is notoriously hard to automatically test and notoriously senstive to the details of the programs being tested, though some testing can be done. We are not yet set up with Sugar to automate any tests.

1. X is running
2. The camera application is able
   1. • to get video
      • display it on the screen
      • The controls work (e.g. contrast, chroma, flip)
3. Touchpad
   1. Test that the capacitive touchpad sensor is working well in relative mode.
   2. Test the resistive sensor, and check the default calibration (details???)
4. USB
   1. insert mouse and test
   2. insert keyboard and test
   3. insert USB devices and check
5. SD
   1. insert SD card assortment and check, from smallest to > 2 gig cards
   2. insert MMC card and verify the system will not read it gracefully
6. Audio
   1. check that sound comes out of the speakers
   2. check that the built in microphone works
   3. check that mic-in works
   4. check that analog in works (to be re-implemented)
7. Battery
   1. check that the battery shows charging or charged
   2. unplug the machine
      • check that the battery shows present and
      • check that the machine does not show it is plugged in
   3. plug in the machine
      • check that the battery shows charging
   4. unplug the battery
      • see that the battery shows not present
   5. reinsert the battery
      • check that the battery shows present and charging again
8. Sugar