Testing checklist
Overview
We are a big believer in automated testing, to the extent that testing can be automated. Computers don't get bored, where as people do (and tend to skip tests). So making it as easy as possible to test our systems has a high priority.
Tinderbox is one such automation tool. In the OLPC tinderbox we can run multiple systems through many, but not all tests, while stressing the system. We reboot the systems, reinstall the image to be tested from scratch, run a set of tests (while loading the system), and repeat. You can see immediately if something has failed, and see where in the testing cycle the failure occurred. Some tinderboxes (e.g. mozilla's) are even able to identify the likely checkin that broke a build being tinderboxed, though that level of automation is currently out of our reach.
Equally important is that tinderboxes can automate performance tests, and arrange to systematically keep track of key performance and/or memory metrics over time, allowing quick identification of performance regressions.
Chris Ball heads OLPC's performance testing work: we're sure he can use help if you want to help in this area; there is always more that can be done!
Automated Tests
* Installation of the BIOS. * Reinstallation of the base software image * Running of CPU burn for a period * Running a video player and dbench on NAND and SD (coming soon). * Rebooting, and repeating.
We need to acquire a programmable power switch, to allow cold reboot tests; right now, we can only perform warm reboots. So far, however, most of our problems have occurred during warm reboots.
Manual Tests
Not all tests can be automated, unfortunately. The testing of user interface software is notoriously hard to automatically test and notoriously senstive to the details of the programs being tested, though some testing can be done. We are not yet set up with Sugar to automate any tests.
# Test that the capacitive touchpad sensor is working well in relative mode. # Test the resistive sensor, and check the calibration (
