Test automation

Resources to aid automated testing of olpc software.

Brainstorming harness architecture

MitchellNCharity 14:01, 4 December 2007 (EST)
overview
 Basically X event scripting and screenshot image analysis, plus
 emulated laptop management, orchestrated by python test control.
x event generation
 tools
  xmacro
  xte
 x server access
  Xephyr, trivial, :N
  on xo, trivial, usual
  on qemu, from outside, need port route (easy),
   and xo X listening to port (how?)
   and shell xauth story (what/which?)
 notes
  may want to postprocess xmacro output
   into xte, for tuning and/or pythonization
   at test time, to do relative location (eg, invite person on screen at x,y)
  xmacro needs a slight pause between motion and click
   might tweak the code to decrease threshold for creating a delay command,
    or hardwire a pause (but pause duration seems likely to be an interesting
    test parameter, so no),
laptop management
 scenarios
  xo-qemu managed emulated xo's
  solitary xo
  coordinating real xo's
   master plus ssh'ed slaves?
 issues
  host needs power and memory to run several qemus
   my 1GB 2GHz only gets me 3ish, and that's with non-standard small memory.
image analysis
 tools
  search - visgrep
  snapshot - import, et al
  masking - ???
   shapes - circle - mask out dynamic center of AP icon - ???
  equality - md5sum
  rough equality - ???
  visual diff - ???
  user mouse-selected region dump tool -  ???
 notes
  crushing to thresholded gray to deal with xo color variation.
  gifs with known palettes to provide "give me foo in colors a,b".
text
 from screenshot
  ocr doesn't seem an option
  cropping to obtain robustness in face of ui changes
 obtain text through means other than screenshot
  gnome accessibility
  watir
harness architecture
 x event generators have few dependencies aside from X 
  so aggregates could be spun off as shell scripts
 python management of laptop, test runs, test results
 test selection is important
  emulated laptops are very expensive to start
  tests aren't fast either
 nose/unittest seems plausible path.  maybe. 
 roles
  regression testing
  repetition for heisenbugs
   needs addiontal capabilities - eg, log capture nnd manipulation
  support activity developers?
test building blocks
 mouse - all mousable state 
 image analysis
  recognizing state (eg, home with frame up)
  extracting key state (eg, mouse over text)
  validating state (eg, compare with known good version)
  determining dynamic parameters (eg, locating non-fixed things)
 senarios
  action/expectation chains
test creation tools
 issues 
  tracking build changes
issues
 screen size in emulation
  Q:is emulation really flaky in Xephyr, or were problems from wrong depth?
  getting 1200x900 in qemu is currently low priority (link ticket, bernie)
  pain to support multiple resolutions
notes
 find mesh circles by number box
 find AP circles by sliver of bottom arc, plus perhaps verification of a
  surrounding masked region.
current state
 was risk exploration.  now architecture planning.
 what exists
  xo-qemu
  kludge python test code
   can recognize activity icons from frame,
    start activites by name, take snapshots.
  visgrep can find xo's in neighborhood view

Resources

• For browser/web tests: http://code.google.com/p/firewatir/ port of http://wtr.rubyforge.org/

See also

• X Window System event scripting
• Tinderbox Testing
• Category:Tests