Testing: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
No edit summary
 
(250 intermediate revisions by 55 users not shown)
Line 1: Line 1:
{{outdated}}
{{OLPC}}
{{OLPC}}
The community and core team work through a slate of '''test issues''' for each major build and release. This page highlights the current tests being run, the testing process in general, and how to contribute your own results or bugs.


{{Latest Releases | firmware = inline | devel = inline | rc = inline | livecd = inline |server = inline }}
Links of Interest:
== Release info ==
* [[Test Group Release Notes]] - Quick info/smoke test on the latest builds. Read this before you bother to load a build
=== Testing for release 11.3.1 ===
* [[OLPC Ship.2 Software Release Notes]] - release notes in progress
Testers are reporting test results on the testing [[Mailing lists|mailing list]] or by [[reporting bugs]].

They are ''not'' using [[Test group release notes]].
=== Testing for release 11.2.0 ===
See [[Test_cases/11.2]] for information about testing the [[11.2.0]] release.

=== Testing for the 10.1.x releases ===
* [[10.1.3/Testing]] for [[:Category:Community testing]] of the [[Release notes/10.1.3|10.1.3 release]]
* [[Smoke_test/10.1.x/1_hour_smoke_test]]

=== Testing for major release 8.2.0 ===
* [[Release_notes/8.2.0#New_features_in_8.2.0|New features in 8.2.0]], on top of [[XO_Base_Features]] of 8.1.1
* [[TestPlan 8.2.0]]
* [[Test cases 8.2.0]], a set of test cases developed in the wiki
* [[TestResults 8.2.0]]

=== Testing for minor releases 8.1.x ===
* Release 8.1.3, [[OLPC SW-ECO 7]], More Amharic support
* Release 8.1.2, [[OLPC SW-ECO 6]], Nand-full, SD card corruption, support of C3 production boards
* Release 8.1.1, [[OLPC SW-ECO 5]], Amharic and Kreyol support
** [[Release_notes/8.1.1|Release notes for 8.1.1]]

=== Testing for release 8.1.0 ===
[[Release notes/8.1.0|Release 8.1.0]] was also known as "Update.1", see [[Update.1 tests]] for tests and [[Testing Update.1 Results]].

==Community testing==
People interested in testing should join the [http://lists.laptop.org/listinfo/testing Testing mailing list], which is also mirrored on an external [http://olpc.396210.n2.nabble.com/Testing-f1469263.html forum].

The awesome New Zealand testers use a very simple [[Activity testing template]].

* [[Systematic testing]]

Test beds:
* [[Creating a testbed]]
* [[Wireless Sparse Testbed]] - Universidade Federal Fluminense, Brazil
* [[Ejabberd resource tests/try_5]] school server jabber testing

''Older'' obsolete community efforts:
* [[Community testing]]
* [[Friends in testing]]
* [[Activity testing project]]
* [[Testing matrix]]
* [[Activity testing template]]

== Links of Interest to Testing ==
* [[Creating new test cases]]
* [[Test Config Notes]] - Links to configuration info that are useful in testing
* [[Test Config Notes]] - Links to configuration info that are useful in testing
* [[Test Group Release Notes]] - Quick info/smoke test on the latest builds. Read this before you bother to load a build
* [[Test meeting Minutes]] - Weekly meetings of interested test people, Mondays 1pm EDT
* [[Upgrading the XO| Different upgrade paths and which to use]]
* [http://www.laptop.org/en/jobs.shtml#QA%20Engineer QA engineer needed] - want to work at our office in Boston to coordinate testing? Apply now.
** Also see Builds of Interest below
* [[Upgrade_VIP|Upgrade for OLPC shipping]]
* [[Multi-hop mesh network in MIT campus]]
* [[Upgrade_B*|Upgrade for G1G1 at Brightstar]]
* [[Networking scenarios]]
* [[Upgrade_Paths| The many different upgrade paths and which to use]]
* [[Simple mesh test (Cerebro)]]
* School Server general info: [[XS]]
** [[Schoolserver Testing]]
* 100 Laptop Testbed:
** [[Collaboration_Network_Testbed]]
** [[Peabody Setup]] - 100 laptop test setup in Peabody, MA
* [[Activation and Developer Keys]] - how to activate or get a developer key
* [[Scenario taxonomy]]
* [[Requirements]] (no longer used)


Equipment specs:
* Infrastructure_AP: Zoom 4400 AP+2 Wireless-G Access Point (Approx. $50 USD)
* Long range Wireless: Wavion’s WS410 Spatially Adaptive Wi-Fi Access Point (Approx. $1600 USD)
* [[XS_Server_Specification#XSX_Specifications|Server HW Specs]]


== Builds of interest==
Check out these new links to test plans and result matrices -- Get involved and help test some activities:
See [[OS images]] for build streams, release candidates, joyride builds, etc.


For latest activities, run Software update in [[Sugar Control Panel]], or download individual [[Activities]].
* [[Activity Testing Project]]
* [[Activity Testing Matrix]]
* [[Activity Testing Template]]


===Older builds and activities ===
* [[Test process sprint]]- Wed, Oct 17th, 2007


* [http://pilgrim.laptop.org/~pilgrim/olpc/streams/update.1/ "Update.1" builds]
* [http://xs-dev.laptop.org/~cscott/olpc/streams/ship.2 "Ship.2" builds]


Activities script/latest activities for 8.1:
=== Test Strategy ===
* http://dev.laptop.org/~erikos/sucrose-activities.py'''
* [http://dev.laptop.org/~erikos/bundles/0.81.4/ Latest 0.81.4 activities] (July 2008)


== Notes and minutes ==
'''Purpose:''' <br>
* [[Testing meetings]] - Weekly meetings of interested test people, usually Thursdays
The purpose of testing is to measure and document the quality of a product.


== General navigation ==
This information, along with the requirements from our customers, will allow us to make an informed decision as to when the product is ready to ship.
[[Test cases 8.2.0]] lists the test cases created for the recent [[Release notes/8.2.0|release 8.2.0]].


Test results are in [[TestResults 8.2.0]]


The {{tl|Community testing}} navigation template is shown below:
'''Plans:''' <br>
{{Community testing}}
The XO software can be [[Emulation|emulated]] on a number of environments, which means many people are helping out with the testing at the unit and module level for software. Our partners who are closest to the hardware and firmware have provided test resources to test in areas that not accessible in the open source community.


<br clear="all" />
There are also many people now from early trials or the developers program who have a laptop and can help out with testing on the hardware. We are working on the idea of a 'Test Activity' that loads test cases directly onto the XO; and sends test results back to olpc. More on that as it materializes.
The navigation template {{tl|Test-nav}} is shown below, it should be updated and many related pages need to be merged:
{{Test-nav}}


== Test cases and test results ==
User-focused test plans are being developed for all the basic features of the laptop, for many of the activities, and for the interaction with the school server, collaboration, connectivity, etc. Please follow this link to read, review, and help create test plans:
Mid-2008 users including Femslade, Charlie, and Kimquirk created test cases for [[8.2.0]] that can be created and modified using a form.
Testers can also use the same form to add testing results, see [[Reporting test results]].


See, e.g. [[Tests/Activity/Write/Public_sharing]] as a sample.
* [[User Stories Testing|User-Focused Test Plans]]


All these test cases are in [[:Category:Test cases]] and most are listed by the queries in [[Test cases 8.2.0]].
Or check out the new pages:
* [[Activity Testing Project]]


In 2007 and early 2008, tests were just wiki pages, e.g. [[Tests/Browse]]. Those older test cases are mostly in [[:Category:Test Plans]] and/or [[:Category:Tests]].


=== Test case management ===
Areas that we need to test in house (due to the use of special equipment and quantities of laptops) include:
* All keyboards. - Bernardo/Kim/Walter
** Keyboard settings preserved across updates.


To add test categories to the pull down menu in [[Form:Test case]], modify
* All languages we can
[[Property:Test category]]
**Mandantory: English, Spanish, Portuguese (Brazil), Arabic for sure; others as possible for update1.


To add stream checkboxes to [[Form:Test case]], modify
* Firmware
[[Property:Build stream]]
** WPA - Richardo/Michailis
** WEP - Richardo/Michailis
** Mesh - upgrade under loaded network - Scott
** Suspend/resume - Wad


To build queries that list test cases, edit [[TestCase Queries]] and copy the <tt><nowiki>{{#ask:&nbsp;...&nbsp;}}</nowiki></tt> markup.
* Scaling/performance
For other examples of queries, see [[Testcase Query Examples]]
** Jabber server scaling - OLPC/Collabora
** Link local presence (Salut) - OLPC/Collabora
** Journal/datastore


Comments at [[Semantic MediaWiki#Issues for Test cases]].
* Upgrades /backward compatibility


== Testing areas ==
:''These "User stories" tests date from 2007-2008''
<!--
These are tests for the builds, materials on them, networks, school servers, and any global synchronization. There is a [[Test activity]] being designed to make testing on an XO simple, and independent of the details of the backend that tracks test results.


Any volunteer who is willing to test should be directed to one single portal: [[Test_issues|Testing]], and s/he can choose which tasks s/he would like to execute. --Chih-yu, Jan 2008
'''If you want to help:'''
-->
# Sign up on the bugtracking system: [http://dev.laptop.org/ Trac]
High level topics for "User stories testing" can be divided into a few major categories:
# Load the latest developer's build: [[autoreinstallation|Install/upgrade]]
# Choose a test plan [[User_Stories_Testing|User-Focused Test Plans]]
# Follow the steps.


=== One Hour Smoke Test ===
If you find a bug you would like to log in trac, please follow these steps:
# Log into Trac: [http://dev.laptop.org/ Trac]
# Use the search to try and determine if your bug has already been reported
# If a bug like yours is there, you can add notes about your instance of this bug.
# If you need to start a new bug, please try to include the following information:
#* A short, but descriptive title (think about what you just tried to search for)
#* What build you were running; both the OS and the firmware if you can
#* Description of what happened (if the title is not enough)
#* Steps to reproduce the bug -- this is really important and will save a lot of time if it is easy to reproduce
#* Add files or logs to help for debugging. See the [[Test Config Notes]] for info on how to turn on logging
#* Choose the component that you think is involved in this bug.
#* Choose the priority -- how important you think it is.
#* Leave the milestone as 'untriaged'. A triage team will review the bugs for priority and add them to the appropriate milestone.


# [[Smoke test]] links to the 1-hour smoke test for different releases, and other smoke tests.
THANKS for your help!


=== Networks ===
# [[Tests/100 laptops]]


=== Views and visualizations ===
Other Test plans can be found here:
*[[802.11s Connectivity Test Plan]] - Connectivity, wifi protocol
*[[Image:B3-B4 OLPC Test Plan.xls]] - Low Level Test Plan
*[[Tinderbox Testing]] - automated testing for each build


#[[Tests/Home view|Home view]]
#[[Tests/Groups view|Groups view]]
#[[Tests/Neighborhood view|Neighborhood view]]
#[[Tests/Frame|Frame]]


<!--
'''Schedule:'''<br>
{|
|-
! Tests !! Results !!
|-
| [[Tests/Mesh view]] || [[Results/Mesh view]]
|-
| [[Tests/Neighborhood view]] || [[Results/Neighborhood view]]
|-
|}
-->


=== Mime Types ===
Please see the official roadmap (from the trac main page) for schedule dates and features for each release.


# [http://www.yolinux.com/TUTORIALS/LinuxTutorialMimeTypesAndApplications.html Mime Types test] from ''yolinux.com''
[https://dev.laptop.org/roadmap Roadmap]
:*[[What should go into Mime Type tests]]


===Battery Life===
== Release Criteria ==
Base line test for battery life. Boot up the machines, fully charge, remove the adapters and see how long the battery lasts.
* [[Battery Results]]
**[[Image:Baseline_battery_test.pdf | Battery Life Test 1 Results]] <br>


=== Regular processes ===
'''Trial-2'''


# [[Tests/Start up and Activation]]
The release criteria for Trial-2 software is based on meeting the expectations of as many as 2000 children and teachers using OLPC products to evaluate whether this is the right product for their school/country. It is expected to be alpha level quality with no serious crash bugs. It is important to be able to demo some of the new features that make our product unique: collaboration both locally and with access to internet, good power consumption, great mechanics/hardware to meet environmental conditions, "open" product -- hardware, firmware, software, activities.
# [[Test Basic Operation]]
# [[Tests/Connectivity and Collaboration]]
# [[Tests/Archive, Backup, Content]]
# [[Tests/Invite]]


=== International Laptops ===
With that in mind, we need to focus on a few good features to demonstrate 'explore', 'express', 'communicate', and 'collaborate' with good quality. This translates into the prioritized feature set for the Trial-2 milestone, which is found here:
# [[Tests/Keyboard mappings]]
# [[Tests/Language settings]]


=== Upgrading ===
'''< http://dev.laptop.org/milestone/Trial-2 >'''
# [[From last stable build]]
# [[Tests/Backward_Comp]]


=== Hardware and firmware ===


# [[Tests/SD card]]
'''Trial-3'''
# [[Tests/SD card acting as a usb drive in card reader]]
# [[Tests/USB stick]]
# [[Tests/Manufacturing Data]]
# [[Tests/Suspend_Resume|Tests/Suspend and resume]]


=== Network/Time Sync ===
Trial-3 needs to meet our beta level expectations. Many more features and activities are demonstrable with collaboration and a solid journal/sugar UI. It is also key that we have activation and upgrades working well. This is the first step in our BitFrost security plan.


# [[Tests/Time sync]]
We will not be adding many new features (perhaps none) between trial-3 and first deployment, so we really need to focus on blocking and high priority bugs associated with basic features and functionality including mesh, connectivity, power management, and activities.
# [[Tests/Network mgr]]

=== SchoolServer ===


===In-house testing (from 2007?) ===
''Old''?

Areas that we need to test in house (due to the use of special equipment and quantities of laptops) include:

* All keyboards. - [[Tests/Keyboard_mappings|Test: Keyboard Settings]]
** Keys match the layouts designed
** Language, keyboard, keymappings match the settings
** All language and keyboard settings are preserved across updates
* Languages - [[Tests/Language_settings|Test: Language Settings]]
* Firmware
** WPA - Richardo/Michailis
** WEP - Richardo/Michailis
** Mesh - upgrade under loaded network - Scott
** Suspend/resume - Wad - [[Tests/Suspend_Resume|Test: Suspend/resume]]
* Scaling/performance
** Jabber server scaling - OLPC/Collabora
** Link local presence (Salut) - OLPC/Collabora
** Journal/datastore
* Upgrades /backward compatibility
* Security


=== Other test plans ===
The prioritized feature list for Trial-3 can be found here:


*[[802.11s Connectivity Test Plan]] - Connectivity, wifi protocol from 2007
'''< http://dev.laptop.org/milestone/Trial-3 >'''
*[[Image:B3-B4 OLPC Test Plan.xls]] - Low level test plan from 2007
*<strike>[[Tinderbox Testing]] - automated testing for each build</strike> (outdated) --[[User:Chihyu|Chihyu]] 13:58, 29 January 2008 (EST)
*[[XMPP collaboration and scalability testing]] - XMPP collaboration and scalability testing
* see [[:Category:Tests]] and [[:Category:Test Plans]] ''TODO: These older test plans should be marked obsolete or merged into test cases.''


==Schedule==


Please see the [http://dev.laptop.org/roadmap official roadmap] (from the trac main page) for schedule dates and features for each release.
'''First Deployment'''


==Test setups==
Release criteria:


This section covers generic ways to set up before running different kinds of tests. (These are not specific test cases, but can be used as shorthand references to common procedures that are used across many test cases.
: ''See [[Release Criteria Trial-2|Release Criteria]]''
* [[Test Config Notes]]
* [[Memory leak testing]]
* [[Olpc-update]]
* [[Clean-install procedure]]


== See also ==


* [http://code.google.com/soc/2008/psf/appinfo.html?csaid=F89A4969030B59E6 GSOC08 OLPC Project Automated Testing]
* [[Test process sprint for 2007 first software release]]- Wed, Oct 17th, 2007


[[Category:Test plans]]
[[Category:Tests]]
[[Category:Developers]]
[[Category:Developers]]
[[Category:Software]]
[[Category:Software]]
[[Category:Hardware]]
[[Category:Hardware]]
[[Category:Subsystems]]
[[Category:Testing]]

Latest revision as of 05:17, 6 August 2013

The contents of this page are considered outdated and some of the information may be stale. Please use information here with caution, or update it.
  This page is monitored by the OLPC team.

The community and core team work through a slate of test issues for each major build and release. This page highlights the current tests being run, the testing process in general, and how to contribute your own results or bugs.


 + Software 
XO
13.2.11 2020-01-29
XS
xs-0.7 2012-02-20


Release info

Testing for release 11.3.1

Testers are reporting test results on the testing mailing list or by reporting bugs.

They are not using Test group release notes.

Testing for release 11.2.0

See Test_cases/11.2 for information about testing the 11.2.0 release.

Testing for the 10.1.x releases

Testing for major release 8.2.0

Testing for minor releases 8.1.x

Testing for release 8.1.0

Release 8.1.0 was also known as "Update.1", see Update.1 tests for tests and Testing Update.1 Results.

Community testing

People interested in testing should join the Testing mailing list, which is also mirrored on an external forum.

The awesome New Zealand testers use a very simple Activity testing template.

Test beds:

Older obsolete community efforts:

Links of Interest to Testing

Equipment specs:

  • Infrastructure_AP: Zoom 4400 AP+2 Wireless-G Access Point (Approx. $50 USD)
  • Long range Wireless: Wavion’s WS410 Spatially Adaptive Wi-Fi Access Point (Approx. $1600 USD)
  • Server HW Specs

Builds of interest

See OS images for build streams, release candidates, joyride builds, etc.

For latest activities, run Software update in Sugar Control Panel, or download individual Activities.

Older builds and activities

Activities script/latest activities for 8.1:

Notes and minutes

General navigation

Test cases 8.2.0 lists the test cases created for the recent release 8.2.0.

Test results are in TestResults 8.2.0

The {{Community testing}} navigation template is shown below:

XO Checkbox


The navigation template {{Test-nav}} is shown below, it should be updated and many related pages need to be merged:

Test cases and test results

Mid-2008 users including Femslade, Charlie, and Kimquirk created test cases for 8.2.0 that can be created and modified using a form. Testers can also use the same form to add testing results, see Reporting test results.

See, e.g. Tests/Activity/Write/Public_sharing as a sample.

All these test cases are in Category:Test cases and most are listed by the queries in Test cases 8.2.0.

In 2007 and early 2008, tests were just wiki pages, e.g. Tests/Browse. Those older test cases are mostly in Category:Test Plans and/or Category:Tests.

Test case management

To add test categories to the pull down menu in Form:Test case, modify Property:Test category

To add stream checkboxes to Form:Test case, modify Property:Build stream

To build queries that list test cases, edit TestCase Queries and copy the {{#ask: ... }} markup. For other examples of queries, see Testcase Query Examples

Comments at Semantic MediaWiki#Issues for Test cases.

Testing areas

These "User stories" tests date from 2007-2008

High level topics for "User stories testing" can be divided into a few major categories:

One Hour Smoke Test

  1. Smoke test links to the 1-hour smoke test for different releases, and other smoke tests.

Networks

  1. Tests/100 laptops

Views and visualizations

  1. Home view
  2. Groups view
  3. Neighborhood view
  4. Frame


Mime Types

  1. Mime Types test from yolinux.com

Battery Life

Base line test for battery life. Boot up the machines, fully charge, remove the adapters and see how long the battery lasts.

Regular processes

  1. Tests/Start up and Activation
  2. Test Basic Operation
  3. Tests/Connectivity and Collaboration
  4. Tests/Archive, Backup, Content
  5. Tests/Invite

International Laptops

  1. Tests/Keyboard mappings
  2. Tests/Language settings

Upgrading

  1. From last stable build
  2. Tests/Backward_Comp

Hardware and firmware

  1. Tests/SD card
  2. Tests/SD card acting as a usb drive in card reader
  3. Tests/USB stick
  4. Tests/Manufacturing Data
  5. Tests/Suspend and resume

Network/Time Sync

  1. Tests/Time sync
  2. Tests/Network mgr

SchoolServer

In-house testing (from 2007?)

Old?

Areas that we need to test in house (due to the use of special equipment and quantities of laptops) include:

  • All keyboards. - Test: Keyboard Settings
    • Keys match the layouts designed
    • Language, keyboard, keymappings match the settings
    • All language and keyboard settings are preserved across updates
  • Languages - Test: Language Settings
  • Firmware
    • WPA - Richardo/Michailis
    • WEP - Richardo/Michailis
    • Mesh - upgrade under loaded network - Scott
    • Suspend/resume - Wad - Test: Suspend/resume
  • Scaling/performance
    • Jabber server scaling - OLPC/Collabora
    • Link local presence (Salut) - OLPC/Collabora
    • Journal/datastore
  • Upgrades /backward compatibility
  • Security

Other test plans

Schedule

Please see the official roadmap (from the trac main page) for schedule dates and features for each release.

Test setups

This section covers generic ways to set up before running different kinds of tests. (These are not specific test cases, but can be used as shorthand references to common procedures that are used across many test cases.

See also

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=Testing&oldid=290836"