How to test an Activity
Introduction
Activity testing is important for volunteer Activity developers so that they'll know what bugs to fix, and for end-users (teachers and students in deployments) so that they know which ones they can use. Activity testing also helps OLPC determine what Activities to ship with G1G1 laptops, and which ones to recommend to large-scale country deployments.
This page is a guide for community members who want to contribute to testing an Activity.
Pick an Activity
The first step is to pick an Activity that you'd like to test. There are many good reasons for picking an Activity, but ultimately, if you like an Activity and want to see it be more widely used, this is the best reason of all. A list of available activities can be found on the Activity page.
(We will be using Analyze as an example in the sections that follow.)
For this tutorial, we will assume that you already have the Activity installed and running on your XO, and that you already know how to use it.
Look at existing resources
The next thing to do is to look at what resources already exist for users, developers, and testers of the Activity. How did you learn about the Activity you are testing, and how did you learn how to use it? If someone wanted to find information about the Activity you are testing, where could they look?
There are usually 4 main sources of information for each Activity:
- The Activity's wiki page (Example: Analyze)
- The Activity's test page (Example: Tests/Activity/Analyze)
- The Activity's Trac component's bugs (Example: Trac query for component: analyze-activity)
- The maintainer's contact info (Example: The Analyze page has a listed contact person.) It is highly recommended that you contact the maintainer at this point to let them know you are testing their Activity - good testing is a partnership between the tester and the developer, and the user community that they serve.
Try searching on the wiki, on the web, and asking in IRC channels to see if any other resources exist for your Activity. Right now you're just trying to get a sense of what is out there. We will address each of these 4 main resources in turn during the sections that follow.
Product and Oracle (wiki page)
In this section, you will be working on the wiki page of the Activity under test to answer two questions:
- What is the product that you're testing? This will almost always be the Activity .xo itself, on a single XO without collaboration (we will focus on these kinds of test in this howto), but there are a few variants on this theme.
- The Activity running on 2 or more XOs, with collaboration.
- The Activity and XO hardware accessories (sensors, peripherals, etc.)
- The Activity and accompanying curriculum/documentation
- What oracle are you using to determine "correct" behavior of your Activity? Some possibilities:
- The description on the wiki page for your Activity. (This may not yet exist; do you have to create it?)
- The specification/features list on the wiki page for your Activity. (Again, these may not yet exist.)
- A heuristic oracle - a set of test inputs with known "correct" outputs (For instance, if you input character X into text box Y, you should see Z displayed. Again, these may not yet exist.)
- Another program that performs a similar function, or which provides input to the Activity under test.
Determine, package, and summarize the product under test.
The wiki page for the Activity under test should have, at the top of the page:
- A one-sentence summary of the product (the Activity). (Example: Analyze)
- Installation instructions, including a link to download an all-in-one package containing all the software, files, etc. someone would need to run the Activity (This is usually the current .xo bundle). (Example: Analyze)
Create these sections if they do not yet exist. This makes it clear what product you are testing.
Find and create user-facing oracles.
The wiki page for the Activity under test should have, under the summary and installation instructions you have just verified or created:
- A features list of what the program is supposed to be able to do. (Example: Analyze)
- A user's guide - instructions on how a user would exercise each of the features in the features list. Screenshots are helpful. This section need not be long; it need only be at the point where a novice user, using your user's guide, would be able to figure out how to exercise each of the features in the features list. Phrases like "each of the columns corresponds to a different measurement in the features list" and "we've shown you how to use feature A with button X; to use features B and C, perform the same actions with buttons Y and Z" are helpful. (Example: [http://wiki.laptop.org/index.php?title=Analyze&oldid=179389#Using_Analyze Analyze)
Create these sections if they do not yet exist. This makes it clear what you expect the Activity to do. (As a bonus, they also make it much easier for others to learn how to use the Activity you are testing!)
It is usually good to talk with the Activity author/maintainer at this point, if you have not yet done so. What did they create the Activity to do? What do they expect to work or not work?
Test strategies (test plan)
This section and the sections below are not yet completed.
- the strategies you're using to organize tests
Black-box testing
- (Optional): Heuristic oracles. This is a list of (Example: None yet. Perhaps you'd like to create one?)
(Optional) White-box testing
- (Optional): White box testing oracles. The features list and user guide oracles are kinds of oracles focus on the user experience. They are a part of black-box testing.
Reporting bugs (trac)
- Why are the bugs you found important? (bug advocacy)
Coverage and completeness (wiki page)
- How "complete" is "good enough"? (There's no such thing as "100% completed testing.")