How to test an Activity
Introduction
Activity testing is important for volunteer Activity developers so that they'll know what bugs to fix, and for end-users (teachers and students in deployments) so that they know which ones they can use. Activity testing also helps OLPC determine what Activities to ship with G1G1 laptops, and which ones to recommend to large-scale country deployments.
This page is a guide for community members who want to contribute to testing an Activity.
This page is not yet complete. We welcome contributions, edits, comments, and testing of these instructions themselves - please walk through steps 1, 2, and 3, which are complete, and leave your comments on the talk page. This will encourage Mchua to finish this page and make it better, faster.
Our goal
Here is the state we're aiming for Activities to get to - remember, the best tester doesn't find the most bugs; the best tester gets the most bugs fixed. Note that it's impossible to completely test something (there is never a way to guarantee something is bug-free), but here's how we can try to get closer to that ideal...
- The Activity's most recent version that works on release 8.2.0 can be installed on an XO with a single click from the Activities page.
- The Activity has an up-to-date (accurate for the latest release) new-user guide on its wiki page (which exists, and is linked to from the Activities page next to the one-click-install link for that Activity). This guide should enable someone who's never used the Activity before to use any of its features. The current version of the guide should have been actually tested on a new user.
- The Activity has an up-to-date (accurate for the latest release) features list of all the things the current version of the Activity can do. The Activity's maintainer should agree with the feature list for the current release (meaning that a tester's been in contact with them for this since the last release).
- The Activity has a test plan current for the latest release that covers the latest feature list. The test plan includes both exploratory and automated tests, has a time estimate for completion, and any tester with an XO should be able to start running through the test plan within 30 minutes of beginning to read the plan page.
- The test plan has been run on the latest Activity version. It has passed. The current status of the test results (pass/fail) is displayed on the Activity's wiki page, and the maintainer knows what the test status is.
- The Activity's wiki page links to the current open bugs and enhancement suggestions on its trac component (which also exists).
- All of the open tickets for that Activity have been verified, have test cases that allow any tester to reproduce a bug and verify it as fixed or not-yet-fixed, have been triaged, and have a developer working on fixing the bug / adding the feature.
- All of the closed tickets have been verified and signed off by a tester who was not the developer who closed the ticket.
- What is this list missing? Please edit and add your thoughts!
Comments from skierpage
- Re 1. install latest version: Running the Software update Sugar Control Panel should upgrade all G1G1 activities to the latest available that works on release 8.2.0. Going to an activity's page and installing the latest may give you an activity that doesn't work on 8.2.0! So, should focus on the links in Activities/G1G1/8.2, not the activity's page. To determine what version of activities you are running on your XO, go to Home view and choose List view.
- Re 1. It seems if you upgrade to 8.2.0 from certain versions, you don't get subscribed to the G1G1 activity group, so you don't get certain activities. Testers should check that their ~olpc/Activities/.groups file contains http://wiki.laptop.org/go/Activities/G1G1 Caryl knows more. (are there bugs on this?)
- If the tester is at least running 8.2-767, this should be as simple as just switching to the Home list view where all Activities installed are shown along with their version number. --Garycmartin 01:06, 8 December 2008 (UTC)
- Re 2. documented activities: http://www.laptop.org/manual describes Record, Browse, Write, Turtle Art. Where should other activities be documented? More pages on flossmanuals?
- Re 4. test plan: Is this a collection of TestCases 8.2.0-style test cases e.g. Tests/Activity/Write/Public sharing, or more like the 2007 Category:Tests e.g. Tests/BlockParty ? Note that if an activity's test plan is just boilerplate like Tests/Activity/Maze, there's no point in demanding one. Testers can run a generic activity smoke test against the activity. CarlKli/cr_lf's Google form that adds to Google spreadsheet is implicitly such a smoke test.
Pick an Activity
The first step is to pick an Activity that you'd like to test. There are many good reasons for picking an Activity, but ultimately, if you like an Activity and want to see it be more widely used, this is the best reason of all.
A list of available activities can be found on the Activity page. We'll be focusing on the activities selected for the 2008 Give One, Get One program; see G1G1 Activity testing.
(We will be using Analyze as an example in the sections that follow.)
For this tutorial, we will assume that you already have the Activity installed and running on your XO, and that you already know how to use it.
"Testing the test instructions" note
While these instructions are under construction, we're trying to keep a list here of who is working on testing what Activity so that there might be less duplication. Please add yourself below - the next check-in is during the Thursday, November 13, 2008 community test meeting.
Participants so far:
Look at existing resources
The next thing to do is to look at what resources already exist for users, developers, and testers of the Activity. How did you learn about the Activity you are testing, and how did you learn how to use it? If someone wanted to find information about the Activity you are testing, where could they look?
There are several main sources of information for each Activity:
- The Activity itself.
- The Activity's wiki page (Example: Analyze)
- The Activity's test case(s) (Example: Tests/Activity/Analyze)
- The Activity's Trac component's bugs (Example: Trac query for component: analyze-activity) This will give you an idea of what known issues already exist with that Activity.
- The maintainer's contact info (Example: The Analyze page has a listed contact person.) It is highly recommended that you contact the maintainer at this point to let them know you are testing their Activity - good testing is a partnership between the tester and the developer, and the user community that they serve.
Try searching on the wiki, on the web, and asking in IRC channels to see if any other resources exist for your Activity. Right now you're just trying to get a sense of what is out there. We will address each of these 4 main resources in turn during the sections that follow, but it is generally a good idea to have these resources open as you work with an Activity or edit its related pages.
Product and Oracle (wiki page)
In this section, you will be working on the wiki page of the Activity under test to answer two questions:
- What is the product that you're testing? This will almost always be the Activity .xo itself, on a single XO without collaboration (we will focus on these kinds of test in this howto), but there are a few variants on this theme.
- The Activity running on 2 or more XOs, with collaboration. (Note: It is recommended that you test Activities in isolation first before testing them with collaboration - the exception to this rule would be Chat, which depends almost entirely on collaboration to function at all.)
- The Activity and XO hardware accessories (sensors, peripherals, etc.)
- The Activity and accompanying curriculum/documentation
- What oracle are you using to determine "correct" behavior of your Activity? Some possibilities:
- The description on the wiki page for your Activity. (This may not yet exist; do you have to create it?)
- The specification/features list on the wiki page for your Activity. (Again, these may not yet exist.)
- A heuristic oracle - a set of test inputs with known "correct" outputs (For instance, if you input character X into text box Y, you should see Z displayed. Again, these may not yet exist.)
- Another program that performs a similar function, or which provides input to the Activity under test.
Determine, package, and summarize the product under test.
The wiki page for the Activity under test should have, at the top of the page:
- A one-sentence summary of the product (the Activity). (Example: Analyze)
- Installation instructions, including a link to download an all-in-one package containing all the software, files, etc. someone would need to run the Activity (This is usually the current .xo bundle). (Example: Analyze)
Create these sections if they do not yet exist. This makes it clear what product you are testing.
Find and create user-facing oracles.
The wiki page for the Activity under test should have, under the summary and installation instructions you have just verified or created:
- A features list of what the program is supposed to be able to do. (Example: Analyze)
- A user's guide - instructions on how a user would exercise each of the features in the features list. Screenshots are helpful. This section need not be long; it need only be at the point where a novice user, using your user's guide, would be able to figure out how to exercise each of the features in the features list. Phrases like "each of the columns corresponds to a different measurement in the features list" and "we've shown you how to use feature A with button X; to use features B and C, perform the same actions with buttons Y and Z" are helpful. (Example: [http://wiki.laptop.org/index.php?title=Analyze&oldid=179389#Using_Analyze Analyze)
Create these sections if they do not yet exist. This makes it clear what you expect the Activity to do. (As a bonus, they also make it much easier for others to learn how to use the Activity you are testing!)
It is usually good to talk with the Activity author/maintainer at this point, if you have not yet done so. What did they create the Activity to do? What do they expect to work or not work?
Test strategies (test plan)
This section is not yet complete.
- the strategies you're using to organize tests
Black-box testing
This section is not yet complete.
- (Optional): Heuristic oracles. This is a list of (Example: None yet. Perhaps you'd like to create one?)
(Optional) White-box testing
This section is not yet complete.
- (Optional): White box testing oracles. The features list and user guide oracles are kinds of oracles focus on the user experience. They are a part of black-box testing.
Reporting test results and bugs
Whether you found bugs or not it's important just to record that somebody did some testing. See Reporting test results (In the community testing nav) for instructions.
Reporting bugs (trac)
This section is not yet complete. See Reporting bugs for now.
- Should we tell testers up front to increase logging level (Attaching Sugar logs to tickets)?
- Why are the bugs you found important? (bug advocacy)
Coverage and completeness (wiki page)
This section is not yet complete.
- How "complete" is "good enough"? (There's no such thing as "100% completed testing.")