Testing: Difference between revisions

From OLPC
Jump to navigation Jump to search
m (fix links, note "Update.1" is also 8.1.0, etc.)
No edit summary
Line 80: Line 80:


====Test Plan====
====Test Plan====

Below is an unformatted (yet) draft. Please excuse its appearance..

Test Plan/Release Criteria for 8.2.0
This is a work-in-progress...
Major test areas/features to be covered are:
Hardware/Firmware (robust behavior, ability to operate with user-oriented (touch-pad, keyboard, etc.) and external devices (mouse, etc.)
Power Management (Power Control & Conservation)
When the lid is closed or when the power button is pressed while Sugar is running, suspend in order to conserve power.
Resume from suspend in response to lid opening or power-button presses
Cut power by holding the power button for several seconds
Power off or reset the machine via the Central XO-person's Palette
Control screen brightness and suspend state from the root command-line
Control screen brightness with the brightness keys in Sugar

Peripherals
Reliably store user data to and retrieve user data from FAT16/VFAT formatted USB Mass Storage devices.
Reliably store user data to and retrieve user data from FAT16/VFAT formatted SD/SDHC devices.
USB ethernet adapters, keyboards, and mice should function similarly to their normal function on other operating systems.
Login via serial consoles.
We do not have strong expectations about the behavior of other peripherals.
Firmware

Developer keys should grant access to OFW.
Should be able to flash any build from OFW.
Should be able to rewrite manufacturing data (e.g. language, write-protect tags) from OFW
Should respond to gamepad Cheat codes
Installation of Software (operating system, Sugar, activities) and firmware. Field upgrades.
W/ internet or preparation: install new activities and library content on a raw OS via the internet or via USB keys.
W/ internet: install new software via 'yum install' or 'rpm -U' on a raw OS via the internet. Data in /home should be PRESERVED.
W/ internet: olpc-update to new versions of the operating system via the internet. Data in /home should be PRESERVED.
W/ preparation: olpc-update to new versions of the operating system via USB. Data in /home should be PRESERVED.
W/ preparation: reflash appropriately signed builds via USB. NO DATA should be preserved.
Access the root account via the virtual terminals or via the Terminal activity (if present.)
Some downstream partners such as Uruguay intend to disable user access to the root account as part of their theft-deterrence plan.
Access to the firmware "ok" prompt w/ developer key, requestable through Browse and installable through the Terminal.
Sugar GUI
TBD
Datastore
Be able to display and edit DOC, ODT, JPEG, PNG, GIF, WAV, OGG, HTML, JS, some SWF, and several other less-well known formats.
We expect NOT to be able to display restricted formats including MP3, MPEG, most AVI, RM, FLV, etc.
Activities
Interact with one activity or "location" at a time
Switch between the current activity and other locations with the view keys or the frame
Switch between activities with Alt-Tab or the activity ring

Start new activities and stop running ones with the frame, ring, or activity toolbar.
Pippy, Record, and TamTam will be able to play and record sounds.
Pippy and Record will be able to take photographs.

Record will be able to record short videos.

Measure will be able to measure signals with or without DC voltage bias.

Volume keys on the keyboard will adjust the speaker volume while Sugar is running.

Control audio volume with the volume keys in sugar
Pictures can be composed with text in a system editor such as Write. (We actually expect Write to do a lot of other things.)
Fancier multimedia like video, audio, and programs can be manipulated in an environment like EToys.

Copy-and-paste media clips between activities

Download media clips into the Journal

Pictures can be composed with text in a system editor such as Write. (We actually expect Write to do a lot of other things.)

Fancier multimedia like video, audio, and programs can be manipulated in an environment like EToys.

Copy-and-paste media clips between activities

Download media clips into the Journal

The section below has to be edited (in progress)…

Installing or removing activities should never change the ability of the system to run.
1 - Start activities at the same speed (including but not limited to all
fructose activities)
2 - Boot to home at the same speed.
3 - Cursor control the same, including number of XOs where it moves
without input or moves without correlation to finger on touchpad
3 - Connect to the same APs in the same time and with the same user
feedback. (e.g. blinks then shows white circle).
4 - Connect to active antennas in the same time and with the same user
feedback (e.g. blinks then shows white circle).
5 - Run all the same activities (can require activity upgrade within
documented bounds). Includes allowing activities to mesh the same way.
6 - Connect to mesh with the same number of XOs in the same time with
the same failure rate.
7 - Save and open files for all activities. Fructose activities at a
minimum otherwise all activities per 5 above. Including when journal has
thousands of entries.
8 - Use the same NAND space.
9 - Always boots up, especially when there is no space on NAND
10 - Open and close all activities per 5 above.
11 - Must support all languages and keyboards previously supported.
"support" means all previously translated strings still work in
activities and sugar. All language types work the same (e.g. RTL)
12 - No new cases where the XO crashes and needs hard reboot.
13 - No new cases where the cursor stops responding (AKA hangs) for more
than 30 seconds.
14 - Sound and video recording and playback must record at the same
rate. Must playback all the same kinds of files. Must playback with the
same quality (assuming all other variables constant, e.g. network,
RAN=M, disk space, etc.)
15 - Browse must render the same web sites as before. See also
http://dev.laptop.org/ticket/7421
16 - USB read, write, mount and unmount must work the same and for all
the same devices.
17 - Mouse and keyboard must connect and work the same
18 - Journal has the same capabilities.
19 - Must use the same or less power for the same user interaction
20 - Must have the same security mechanisms and be as secure as previous
releases.
21 - Must be installable in the factory and upgradeable from 656, 703
and 708 via USB or network.
--------
Collaboration/Connectivity
Between XOs with simple mesh (associate with other XO or Active-Antenna 802.11s meshes).
Between XOs with a school server with an active antenna and an access point (associate with 802.11 b/g access points with or without WEP and WPA1 encryption).
In a mixed environment.
With non-XO machines
Security
Laptops should travel through the delivery chain unactivated.
Laptops should not boot until they are activated.
Can be activated wirelessly (with a school server) (not recently tested)

Can be activated via USB.
Performance/scalability
TBD
Localization, including correctness of language-specific icon-related text
Keymaps are correct or are very close to correct for US/International, Amharic, Spanish, Portuguese, and Kreyól keyboards.
Translations of system strings need help!
User documentation - inside XO and online (wiki) available
TBD
Maintaining history
List, search, and filter records of actions on the system by time
filter records of actions by title

----------------------------------------


This is still work-in-progress...
This is still work-in-progress...

Revision as of 04:24, 31 July 2008

  This page is monitored by the OLPC team.

The community and core team work through a slate of test issues for each major build and release. This page highlights the current tests being run, the testing process in general, and how to contribute your own results or bugs.

For the latest tests, see Update.1 tests.

Notes and minutes

General navigation

Many related pages need to be merged:

Activity testing project | Testing matrix | Activity testing guide | Activity testing template +/-

To merge: Testing | Testing ideas | Friends in testing


See also: Sugar Labs testing | the Testing mailing list


Release info

Older notes

Links of Interest

Equipment specs:

  • Infrastructure_AP: Zoom 4400 AP+2 Wireless-G Access Point (Approx. $50 USD)
  • Long range Wireless: Wavion’s WS410 Spatially Adaptive Wi-Fi Access Point (Approx. $1600 USD)
  • Server HW Specs (Approx. $3000 USD)

Check out these new links to test plans and result matrices -- Get involved and help test some activities:

Community Testing:

Builds of Interest:

Activities script/latest activities:

Testing teams

The Activities on the XO have been developed by community members. Many of them are not well documented and there may not be much information on how to use them; so this testing can be described as 'Descriptive Testing', where you are helping to provide and/or correct the wiki pages that describe how an activity works. If you want to help, please refer to the Community Testing page for more details.

Testing at OLPC

Test Plan

Below is an unformatted (yet) draft. Please excuse its appearance..

Test Plan/Release Criteria for 8.2.0 This is a work-in-progress... Major test areas/features to be covered are: Hardware/Firmware (robust behavior, ability to operate with user-oriented (touch-pad, keyboard, etc.) and external devices (mouse, etc.) Power Management (Power Control & Conservation) When the lid is closed or when the power button is pressed while Sugar is running, suspend in order to conserve power. Resume from suspend in response to lid opening or power-button presses Cut power by holding the power button for several seconds Power off or reset the machine via the Central XO-person's Palette Control screen brightness and suspend state from the root command-line Control screen brightness with the brightness keys in Sugar

Peripherals Reliably store user data to and retrieve user data from FAT16/VFAT formatted USB Mass Storage devices. Reliably store user data to and retrieve user data from FAT16/VFAT formatted SD/SDHC devices. USB ethernet adapters, keyboards, and mice should function similarly to their normal function on other operating systems. Login via serial consoles. We do not have strong expectations about the behavior of other peripherals.

Firmware

Developer keys should grant access to OFW. Should be able to flash any build from OFW. Should be able to rewrite manufacturing data (e.g. language, write-protect tags) from OFW Should respond to gamepad Cheat codes

Installation of Software (operating system, Sugar, activities) and firmware. Field upgrades. W/ internet or preparation: install new activities and library content on a raw OS via the internet or via USB keys. W/ internet: install new software via 'yum install' or 'rpm -U' on a raw OS via the internet. Data in /home should be PRESERVED. W/ internet: olpc-update to new versions of the operating system via the internet. Data in /home should be PRESERVED. W/ preparation: olpc-update to new versions of the operating system via USB. Data in /home should be PRESERVED. W/ preparation: reflash appropriately signed builds via USB. NO DATA should be preserved. Access the root account via the virtual terminals or via the Terminal activity (if present.) Some downstream partners such as Uruguay intend to disable user access to the root account as part of their theft-deterrence plan. Access to the firmware "ok" prompt w/ developer key, requestable through Browse and installable through the Terminal. Sugar GUI TBD Datastore

Be able to display and edit DOC, ODT, JPEG, PNG, GIF, WAV, OGG, HTML, JS, some SWF, and several other less-well known formats. 

We expect NOT to be able to display restricted formats including MP3, MPEG, most AVI, RM, FLV, etc. Activities Interact with one activity or "location" at a time Switch between the current activity and other locations with the view keys or the frame Switch between activities with Alt-Tab or the activity ring

Start new activities and stop running ones with the frame, ring, or activity toolbar.

Pippy, Record, and TamTam will be able to play and record sounds. Pippy and Record will be able to take photographs.

Record will be able to record short videos.

Measure will be able to measure signals with or without DC voltage bias.

Volume keys on the keyboard will adjust the speaker volume while Sugar is running.

Control audio volume with the volume keys in sugar

Pictures can be composed with text in a system editor such as Write. (We actually expect Write to do a lot of other things.) Fancier multimedia like video, audio, and programs can be manipulated in an environment like EToys.

Copy-and-paste media clips between activities

Download media clips into the Journal

Pictures can be composed with text in a system editor such as Write. (We actually expect Write to do a lot of other things.)

Fancier multimedia like video, audio, and programs can be manipulated in an environment like EToys.

Copy-and-paste media clips between activities

Download media clips into the Journal

The section below has to be edited (in progress)…

Installing or removing activities should never change the ability of the system to run. 1 - Start activities at the same speed (including but not limited to all fructose activities) 2 - Boot to home at the same speed. 3 - Cursor control the same, including number of XOs where it moves without input or moves without correlation to finger on touchpad 3 - Connect to the same APs in the same time and with the same user feedback. (e.g. blinks then shows white circle). 4 - Connect to active antennas in the same time and with the same user feedback (e.g. blinks then shows white circle). 5 - Run all the same activities (can require activity upgrade within documented bounds). Includes allowing activities to mesh the same way. 6 - Connect to mesh with the same number of XOs in the same time with the same failure rate. 7 - Save and open files for all activities. Fructose activities at a minimum otherwise all activities per 5 above. Including when journal has thousands of entries. 8 - Use the same NAND space. 9 - Always boots up, especially when there is no space on NAND 10 - Open and close all activities per 5 above. 11 - Must support all languages and keyboards previously supported. "support" means all previously translated strings still work in activities and sugar. All language types work the same (e.g. RTL) 12 - No new cases where the XO crashes and needs hard reboot. 13 - No new cases where the cursor stops responding (AKA hangs) for more than 30 seconds. 14 - Sound and video recording and playback must record at the same rate. Must playback all the same kinds of files. Must playback with the same quality (assuming all other variables constant, e.g. network, RAN=M, disk space, etc.) 15 - Browse must render the same web sites as before. See also http://dev.laptop.org/ticket/7421 16 - USB read, write, mount and unmount must work the same and for all the same devices. 17 - Mouse and keyboard must connect and work the same 18 - Journal has the same capabilities. 19 - Must use the same or less power for the same user interaction 20 - Must have the same security mechanisms and be as secure as previous releases. 21 - Must be installable in the factory and upgradeable from 656, 703 and 708 via USB or network.


Collaboration/Connectivity Between XOs with simple mesh (associate with other XO or Active-Antenna 802.11s meshes). Between XOs with a school server with an active antenna and an access point (associate with 802.11 b/g access points with or without WEP and WPA1 encryption). In a mixed environment. With non-XO machines Security Laptops should travel through the delivery chain unactivated. Laptops should not boot until they are activated. Can be activated wirelessly (with a school server) (not recently tested)

Can be activated via USB. Performance/scalability TBD Localization, including correctness of language-specific icon-related text Keymaps are correct or are very close to correct for US/International, Amharic, Spanish, Portuguese, and Kreyól keyboards. Translations of system strings need help! User documentation - inside XO and online (wiki) available TBD Maintaining history List, search, and filter records of actions on the system by time filter records of actions by title


This is still work-in-progress...

Major test areas to be covered are:

Hardware (robust behavior, ability to operate with user-oriented (touch-pad, keyboard,etc.) and external devices (mouse, etc.)

Installation of Software (operating system, Sugar, activities) and firmware.

Field upgrades

Sugar GUI

Datastore

Activities

Power management

Collaboration/Connectivity

  • Between XOs with simple mesh
  • Between XOs with a school server with an active antenna and an access point
  • In a mixed environment

Security

Performance/scalability

Localization

Correctness of language-specific icon-related text

User documentation - inside XO and online (wiki) available

Interoperability with non-XO machines

To report results from a set of test cases, add a line to the appropriate table in the test case pages below with a quick summary; and include details, including setup and test team, on the talk page.

Items to Test

Areas that we need to test in house (due to the use of special equipment and quantities of laptops) include:

  • All keyboards. - Test: Keyboard Settings
    • Keys match the layouts designed
    • Language, keyboard, keymappings match the settings
    • All language and keyboard settings are preserved across updates
  • Languages - Test: Language Settings
  • Firmware
    • WPA - Richardo/Michailis
    • WEP - Richardo/Michailis
    • Mesh - upgrade under loaded network - Scott
    • Suspend/resume - Wad - Test: Suspend/resume
  • Scaling/performance
    • Jabber server scaling - OLPC/Collabora
    • Link local presence (Salut) - OLPC/Collabora
    • Journal/datastore
  • Upgrades /backward compatibility
  • Security


Other Test plans:


Schedule:

Please see the official roadmap (from the trac main page) for schedule dates and features for each release.

Also See

GSOC08 OLPC Project Automated Testing