Test automation


Jump to: navigation, search

Resources to aid automated testing of olpc software.



olpc-xo-qemu makes it easier to create and control QEMU emulated laptops. It can also control real XO's, with the same interface.

X event scripting

See X Window System event scripting.

OCR of screenshots

GOCR (see Optical character recognition) works fairly well with screenshots. In English at least. Recognition of text in Write is quite good. In Browse, rather noisy. In Calculate, you get noisy results if you do the entire screen, but if you crop out the number display field, then the results are quite good.

So you can indeed use OCR for testing automation, but for each use, you need to see how well it works, and perhaps tweak it.

I expect one could do non-english Latin alphabets as well. I've not tried it. It might require switching to ocre. Arabic, Thai, Chinese, etc, are not feasible with open-source OCR. Russian might be, but it wouldn't be turn-key.

Old brainstorming of test harness architecture

MitchellNCharity 14:01, 4 December 2007 (EST)
 Basically X event scripting and screenshot image analysis, plus
 emulated laptop management, orchestrated by python test control.
x event generation
 x server access
  Xephyr, trivial, :N
  on xo, trivial, usual
  on qemu, from outside, need port route (easy),
   and xo X listening to port (how?)
   and shell xauth story (what/which?)
  may want to postprocess xmacro output
   into xte, for tuning and/or pythonization
   at test time, to do relative location (eg, invite person on screen at x,y)
  xmacro needs a slight pause between motion and click
   might tweak the code to decrease threshold for creating a delay command,
    or hardwire a pause (but pause duration seems likely to be an interesting
    test parameter, so no),
laptop management
  xo-qemu managed emulated xo's
  solitary xo
  coordinating real xo's
   master plus ssh'ed slaves?
  host needs power and memory to run several qemus
   my 1GB 2GHz only gets me 3ish, and that's with non-standard small memory.
image analysis
  search - visgrep
  snapshot - import, et al
  masking - ???
   shapes - circle - mask out dynamic center of AP icon - ???
  equality - md5sum
  rough equality - ???
  visual diff - ???
  user mouse-selected region dump tool -  ???
  crushing to thresholded gray to deal with xo color variation.
  gifs with known palettes to provide "give me foo in colors a,b".
 from screenshot
  ocr doesn't seem an option
  cropping to obtain robustness in face of ui changes
 obtain text through means other than screenshot
  gnome accessibility
harness architecture
 x event generators have few dependencies aside from X 
  so aggregates could be spun off as shell scripts
 python management of laptop, test runs, test results
 test selection is important
  emulated laptops are very expensive to start
  tests aren't fast either
 nose/unittest seems plausible path.  maybe. 
  regression testing
  repetition for heisenbugs
   needs addiontal capabilities - eg, log capture nnd manipulation
  support activity developers?
test building blocks
 mouse - all mousable state 
 image analysis
  recognizing state (eg, home with frame up)
  extracting key state (eg, mouse over text)
  validating state (eg, compare with known good version)
  determining dynamic parameters (eg, locating non-fixed things)
  action/expectation chains
test creation tools
  tracking build changes
 screen size in emulation
  Q:is emulation really flaky in Xephyr, or were problems from wrong depth?
  getting 1200x900 in qemu is currently low priority (link ticket, bernie)
  pain to support multiple resolutions
 find mesh circles by number box
 find AP circles by sliver of bottom arc, plus perhaps verification of a
  surrounding masked region.
current state
 was risk exploration.  now architecture planning.
 what exists
  kludge python test code
   can recognize activity icons from frame,
    start activites by name, take snapshots.
  visgrep can find xo's in neighborhood view


See also

Personal tools
  • Log in
  • Login with OpenID
About OLPC
About the laptop
About the tablet
OLPC wiki