How to test an Activity
Introduction
Activity testing is important for volunteer Activity developers so that they'll know what bugs to fix, and for end-users (teachers and students in deployments) so that they know which ones they can use. Activity testing also helps OLPC determine what Activities to ship with G1G1 laptops, and which ones to recommend to large-scale country deployments.
This page is a guide for community members who want to contribute to testing an Activity.
This page is not yet complete. We welcome contributions, edits, comments, and testing of these instructions themselves - please walk through steps 1, 2, and 3, which are complete, and leave your comments on the talk page. This will encourage Mchua to finish this page and make it better, faster.
Our goal
Here is the state we're aiming for Activities to get to - remember, the best tester doesn't find the most bugs; the best tester gets the most bugs fixed. Note that it's impossible to completely test something (there is never a way to guarantee something is bug-free), but here's how we can try to get closer to that ideal...
- The Activity's most recent version can be installed on an XO with a single click from the Activities page.
- The Activity has an up-to-date (accurate for the latest release) new-user guide on its wiki page (which exists, and is linked to from the Activities page next to the one-click-install link for that Activity). This guide should enable someone who's never used the Activity before to use any of its features. The current version of the guide should have been actually tested on a new user.
- The Activity has an up-to-date (accurate for the latest release) features list of all the things the current version of the Activity can do. The Activity's maintainer should agree with the feature list for the current release (meaning that a tester's been in contact with them for this since the last release).
- The Activity's wiki page links to the current open bugs and enhancement suggestions on its trac component (which also exists).
- All of the open tickets for that Activity have been verified, have test cases that allow any tester to reproduce a bug and verify it as fixed or not-yet-fixed, have been triaged, and have a developer working on fixing the bug / adding the feature.
- All of the closed tickets have been verified and signed off by a tester who was not the developer who closed the ticket.
- What is this list missing? Please edit and add your thoughts!
Pick an Activity
The first step is to pick an Activity that you'd like to test. There are many good reasons for picking an Activity, but ultimately, if you like an Activity and want to see it be more widely used, this is the best reason of all. A list of available activities can be found on the Activity page.
(We will be using Analyze as an example in the sections that follow.)
For this tutorial, we will assume that you already have the Activity installed and running on your XO, and that you already know how to use it.
"Testing the test instructions" note
While these instructions are under construction, we're trying to keep a list here of who is working on testing what Activity so that there might be less duplication. Please add yourself below - the next check-in is during the Thursday, November 13, 2008 Community test meeting.
Participants so far:
Look at existing resources
The next thing to do is to look at what resources already exist for users, developers, and testers of the Activity. How did you learn about the Activity you are testing, and how did you learn how to use it? If someone wanted to find information about the Activity you are testing, where could they look?
There are several main sources of information for each Activity:
- The Activity itself.
- The Activity's wiki page (Example: Analyze)
- The Activity's test case(s) (Example: Tests/Activity/Analyze)
- The Activity's Trac component's bugs (Example: Trac query for component: analyze-activity) This will give you an idea of what known issues already exist with that Activity.
- The maintainer's contact info (Example: The Analyze page has a listed contact person.) It is highly recommended that you contact the maintainer at this point to let them know you are testing their Activity - good testing is a partnership between the tester and the developer, and the user community that they serve.
Try searching on the wiki, on the web, and asking in IRC channels to see if any other resources exist for your Activity. Right now you're just trying to get a sense of what is out there. We will address each of these 4 main resources in turn during the sections that follow, but it is generally a good idea to have these resources open as you work with an Activity or edit its related pages.
Product and Oracle (wiki page)
In this section, you will be working on the wiki page of the Activity under test to answer two questions:
- What is the product that you're testing? This will almost always be the Activity .xo itself, on a single XO without collaboration (we will focus on these kinds of test in this howto), but there are a few variants on this theme.
- The Activity running on 2 or more XOs, with collaboration. (Note: It is recommended that you test Activities in isolation first before testing them with collaboration - the exception to this rule would be Chat, which depends almost entirely on collaboration to function at all.)
- The Activity and XO hardware accessories (sensors, peripherals, etc.)
- The Activity and accompanying curriculum/documentation
- What oracle are you using to determine "correct" behavior of your Activity? Some possibilities:
- The description on the wiki page for your Activity. (This may not yet exist; do you have to create it?)
- The specification/features list on the wiki page for your Activity. (Again, these may not yet exist.)
- A heuristic oracle - a set of test inputs with known "correct" outputs (For instance, if you input character X into text box Y, you should see Z displayed. Again, these may not yet exist.)
- Another program that performs a similar function, or which provides input to the Activity under test.
Determine, package, and summarize the product under test.
The wiki page for the Activity under test should have, at the top of the page:
- A one-sentence summary of the product (the Activity). (Example: Analyze)
- Installation instructions, including a link to download an all-in-one package containing all the software, files, etc. someone would need to run the Activity (This is usually the current .xo bundle). (Example: Analyze)
Create these sections if they do not yet exist. This makes it clear what product you are testing.
Find and create user-facing oracles.
The wiki page for the Activity under test should have, under the summary and installation instructions you have just verified or created:
- A features list of what the program is supposed to be able to do. (Example: Analyze)
- A user's guide - instructions on how a user would exercise each of the features in the features list. Screenshots are helpful. This section need not be long; it need only be at the point where a novice user, using your user's guide, would be able to figure out how to exercise each of the features in the features list. Phrases like "each of the columns corresponds to a different measurement in the features list" and "we've shown you how to use feature A with button X; to use features B and C, perform the same actions with buttons Y and Z" are helpful. (Example: [http://wiki.laptop.org/index.php?title=Analyze&oldid=179389#Using_Analyze Analyze)
Create these sections if they do not yet exist. This makes it clear what you expect the Activity to do. (As a bonus, they also make it much easier for others to learn how to use the Activity you are testing!)
It is usually good to talk with the Activity author/maintainer at this point, if you have not yet done so. What did they create the Activity to do? What do they expect to work or not work?
Test strategies (test plan)
This section is not yet complete.
- the strategies you're using to organize tests
Black-box testing
This section is not yet complete.
- (Optional): Heuristic oracles. This is a list of (Example: None yet. Perhaps you'd like to create one?)
(Optional) White-box testing
This section is not yet complete.
- (Optional): White box testing oracles. The features list and user guide oracles are kinds of oracles focus on the user experience. They are a part of black-box testing.
Reporting test results and bugs
Whether you found bugs or not it's important just to record that somebody did some testing. See Reporting test results (In the community testing nav) for instructions.
Reporting bugs (trac)
This section is not yet complete. See Reporting bugs for now.
- Why are the bugs you found important? (bug advocacy)
Coverage and completeness (wiki page)
This section is not yet complete.
- How "complete" is "good enough"? (There's no such thing as "100% completed testing.")