Talk:Test cases 8.2.1

Notes from 2008-12-23

These are rough notes from a conversation with Ed McNierney (8.2.1 release manager) and Kim Quirk (VP of Testing & Support), with Frances Hopkins (OLPC QA for 8.2.0) and Michael Stone (8.2.0 release manager, 8.2.1 developer) joining in towards the end. They're probably incomprehensible to most people right now, but please ask if you have questions or want clarifications and I'm happy to edit. Mchua 16:45, 24 December 2008 (UTC)

Ticket testing
---------------
- Refers to "are the blockers for this release closed?"
- Important, and currently the only thing in the test plan
- There should be more kinds of testing, though (notably, smoke testing
  and also examining fixes to these tickets for areas of risk that we need
  to regression test against).

Connectivity testing
----------------------

- degrading connectivity: the quickest check is to look at Neighborhood
  for 2 XOs that should be visible to each other; if the Neighborhoods of
  both XOs look different, or strange things (Activities without anyone
  sharing them, XO-people gathered around a nonexistent Activity) appear,
  it's a likely sound of connectivity degradation and further tests should
  be performed.
- when doing mesh testing, you want RF isolation; nothing else should show
  up on Neighborhood view.
- mesh testing is incredibly noisy (in the RF sense) because it's a multicast
  spew - for N laptops in a mesh, each laptop has to continuously ping N-1
  other laptops.
- there are two kinds of communications that create noise with XOs: "Are you
  there?" presence pings, and "We're sharing an activity" collaboration.
  (aka "what are you doing?" pings.)
- when laptops are in a simple mesh, presence noise and activity noise are
  high because each XO has to ping N-1 other XOs for it.
- when laptops are on an AP, the presence noise goes down (each XO now has
  to keep track of 1 point - the AP - for presence) but collaboration
  (jabber) is still XO-to-XO so the noise is high.
- when laptops are on an XS, presence and collaboration are both done through
  the single point of the XS, so noise is low; this is why we can get
  many laptops collaborating on an XS.

Connectivity tests pass if they work for these numbers:
10 laptops in a simple mesh
20 laptops on a single AP
50 laptops on a XS

- Noise notes: most non-XO laptops start trying to connect with 802.11a, XOs do b/g. 
- XOs default to trying to come up on channel 1 when they're turned on. This means
  it is a bad idea to turn on 50 XOs at once (when trying to run the 50 XOs on an XS
  tests, for example) because they'll all come up multicasting on channel 1 and
  everyone attempting to connect to wifi around you will be rather upset by their
  newfound inability to do so. Bring XOs up in small groups (5 is a good number)
  and switch them to another channel; connect them to an XS (stop the multicast spew
  and then go for the next group.

When running connectivity tests, it doesn't matter what AP (hardware) you ran them with,
so long as you document it the AP you used.

Do make sure that wpa, wep, wpa2, etc. work.

Connectivity testing is particularly intermittent. We need to create a standard for
how many/frequently/often connectivity tests have to pass for connectivity to be
declared "good."

Terminology
------------

Ed's definition of a beta release: It's done, but there are probably bugs.
In other words, there's nothing known wrong, but we probably don't know what's wrong.

firmware testing
-------------------
Must test all 4 options for:
signed, unsigned
secure, unsecure

Also look at upgrade paths for what is out there
(what are customers running? find out!)

improving the release testing process
----- --------------------------------

collect more test cases. 8.2.0 was the first time we did this at all
status reports: some easy big green bar indicator of "how close we are
to done" at all times

comumunity testers
--------------------

- enabling them is a lot of work (it's an investment).

Things they can do that 1cc can't:

- have many more man-hours that we do
- environmental permutations of both hardware and RF environment

Things 1cc can do that they can't easily (yet):

- access to large-scale testbeds (20, 50 XOs, an XS)
- access to deployments

acceptance testing
--------------------

Making deployment-specific test case stubs may be a good idea.
(In other words, a test which passes only if it's, say, Peru that
does the testing.)

We should try to get deployments to do this.
Highest priority: Uruguay.
Peru and Mongolia may be able to help. (Peru has a testbed,
Mongolia has an excellent and excited engineer.)
Oceania may be a good place to try grassroots acceptance testing.

Questions/Todo (for mchua)
------------------------------
- gather more todos from these notes
- how can you tell what channel an AP is on? (In general, I need to learn
  a lot more about how wireless works. --Mchua)
- set up an RF monitor. Reuben might have one.
- talk with Joe about testbed usage
- talk with Reuben about XS usage
- acceptance testing by deployments: have those conversations