OLPC Rochester, NY/Usability testing class project

Top page for class project.

NOTE: The contents of this page are not set in stone, and are subject to change! This page is a draft in active flux ... Please leave suggestions on the talk page.

Testing project abstract

The OLPC (One Laptop per Child project) provides school-aged children in developing countries with specially-designed learning machines (the XO Laptop). These machines are intended to allow students access to new tools as well as a window to the rest of the world. Although the XO is in mass production and has been widely distributed, the open-source software at its core is under continuous development. Considering the target user-group of technologically inexperienced children, we will examine the ease of use of the XO's Sugar graphical user interface for elementary school children by focusing on the built-in Write activity (application/program). The OLPC’s ideal of collaboration will be studied by observing how the children interact and perform in this shared activity which offers multiple simultaneous cursors for synchronous editing. This is a relatively unstudied social computing capability.

Testing will be executed on site at the Wheatland-Chili Central School District elementary school on multiple dates. An introduction to our team and our intentions will be carried out by the school superintendent, class teacher, or other school faculty/staff member. The tests will then be conducted in two stages, using different participants in each stage. The initial test format will be almost exclusively observational—allowing the students to freely interact with the XO laptops and taking note of their various experiences. The second phase of testing will involve a new set of students, studied in pairs as they interact on separate XO’s using the Write activity.

Following a short period of familiarizing each student with the basic functions of the XO laptop, we will wirelessly network both units so that the Write application is shared between them. Each student will then be provided with a clear, large-print document containing approximately 6 to 8 lines of simple text (we intend to use verses of a Dr. Seuss book). Different lines of text will be one of two colors (e.g., RED vs BLUE) and each student will be assigned one of the two colors. Their task will be to re-create the provided verse using the shared Write activity. Each child will be responsible only for typing in text displayed in their assigned color. We will observe the manner in which they perform the task (i.e., waiting and taking turns or working simultaneously). We may also alter portions of the text, such as adding highlights or bold letters, to measure their success in using some of the features in Write.

Both stages of testing will involve video recording (including voice) of the sessions. One to three cameras will document how the students interact and work with each other and the XO units. We will attempt to restrict and focus all visual recording to the students’ physical interactions with the XO (on the hands or over-the-shoulder at the screen activity). The video footage will then be reviewed and edited or altered in such a way that any instances of a participant’s face shall be censored or removed to protect their privacy.

We expect the observational test phase to last approximately 15 minutes per “free play” session. The writing task phase is expected to last approximately 5 – 20 minutes, or less if the task is fully completed. In the event that a pair of students encounters difficulty completing the task, we will conclude the testing after a cap of 25 minutes. At the conclusion of each session, the students will return to their normal instruction in the classroom.

We are considering the distribution of individually bagged pretzels or another approved snack as a reward for participation. After all testing is completed, we are also considering providing the class with a collection of color printouts containing the various verses used in the test session. Each page of verse will include an area where the participating students can write their names in the color they were assigned. All of the printouts could be arranged and combined to reveal a larger portion of the work it was selected from (Dr. Seuss) and the participating children can share with and show others what they accomplished by working together using the XO laptops.

Types of data to be collected

• Demographics: Child’s age, sex, native and/or other languages, and ethnicity
• Details concerning child’s access to computers/laptops (at home, at school, etc)
• Child’s computer usage habits (frequency)
• Whether or not the child owns his or her own computer
• A measure of the child’s computer experience by verbal surveys (answer forms include choices, rating scales, open-ended, etc.) made less formal for children by verbal questioning as if a question and answer session
• Observations of the child’s experience with a Windows XP system (the completion of 3-4 pre-determined tasks to establish an experience baseline)
• Video and audio taping of the child’s activities and interactions with the XO laptop alone or with another child

Test plan outline

After permission from the school we intend to use as a testing site is obtained, and a letter of support from them is received, all required materials will be submitted to the RIT's Institutional Review Board for approval.

We will coordinate with the school superintendent (Thomas Gallagher), technology coordinator (Ted Sakshaug), and classroom teacher(s) to set up appropriate test days and times for us to conduct our studies. This information will be outlined in a Communication of Intent Form, which will be given to the teacher to distribute among potential student participants. In addition, this package will contain information concerning the One Laptop per Child (OLPC) program and informed consent and assent forms. The packet is to be taken home by any interested students for parental review and approval (signing of both the informed consent and assent forms to return to school for our collection).

On the day(s) of testing, we shall arrive on site at the school and be introduced to the children and class by the superintendent and/or class teacher. Following a brief greeting and any clarification of our purpose by our group, we will collect and review informed consent and assent forms. Those who do not have these forms, or do not wish to participate shall be excluded from further activity.

The remaining qualifying children will complete a simple four-question screener that verifies their ability to participate in the testing. Those who can further continue will be randomly selected and paired up for the testing sessions. Each pair of students will participate in the observational study in an area designated as appropriate and agreed upon by the class teacher, school superintendent, and test group.

There will be two types of tests conducted. In the first, the child participant will be given access to an XO laptop unit and will be encouraged to freely interact with and use it as they please. In the second test, a different set of students not yet exposed to the XO laptops will be asked to work in pairs on a writing program that is wirelessly connected and shared between two units. They will be given a short excerpt from a book or reading (Dr. Seuss) and asked to re-create it on their XO unit. In both types of tests, the children’s interactions with the XO laptop and with one another will be observed and recorded.

Data shall be collected by video/audio recording of the sessions, taking notes of observations, and by informal verbal interviews. Both before and after each of the two types of test sessions, the test conductors will engage the student participants, one-on-one, in verbal pre- and post-test surveys. Some of the questions will have choices, some will contain a rating scale, and some will be open-ended. The test conductor will read the questions and record the answers for the child.

At the conclusion of all testing, the test group will address any final questions and concerns from the school teacher, staff, and children. If we decide to distribute individually bagged snacks, approved by the school, as a participation reward they will be given at the end of each test session. Any other items we intend to distribute shall be given at this time before our departure.

We are considering distributing a copy of the students’ final collaborative work to them as a sense of accomplishment and also as a means for them to explain to their class, teacher, and/or parents what they achieved with one another and the XO laptop.

Protocol summary

1. Students will be informed by their teacher or superintendent of our intentions and eventual visit to their classroom.
2. Informational packet to parents is distributed in class for children to take home (includes: letter of intent, OLPC overview information sheet, parental informed consent form, child assent form).
3. Arrival at the testing site (school).
4. Introduction to student body of participants by teacher or superintendent.
5. Brief greeting and description of purpose by testing group members.
6. Informed consent/ child assent form collection and review. Only children with approved forms may participate any further.
7. Participant screeners distributed, completed, and collected. Quick review for the exclusion of incompatible participants.
8. Random selection and pairing of students by number assignment / lot drawing.
9. Pre-trial survey/interview conducted by one tester to one child (a pair at a time).
10. Commencement of trials (two students at a time) for task-less, free interaction observational data, followed by a brief verbal post-test survey.
11. Pre-trial survey/interview conducted by one tester to one child (a pair at a time).
12. Commencement of trials (two students at a time) for task-based write program collaboration, followed by brief verbal post-test survey.
13. Conclusion, thanks, addressing of questions and concerns from the class and participants.

Task list

Test Session 1: “Free-Play” Interaction

Students will be given an XO laptop and informed that they are free to “try it out” and use it however they’d like. They will be encouraged to speak out loud as they use it because we’re interested in knowing what they think about the device as they use it. Minimal support will be given to them as they interact with the XO in order to keep the session productive. Instances of this include helping them open the XO, if necessary, and assisting them with any software difficulties that have halted their activity. Test conductors will attempt to be judicious and consistent when these incidents arise.

Test Session 2: Writing task

1. Each student in a randomly-selected and arranged pair will be given an XO laptop and briefly familiarized with the main, basic functions of the device (touchpad use, left-click/button pushing, etc.).
2. The students will be seated across from one another at a small table.
3. Each student will be designated one of two colors (used on the printout of the text we have chosen for the study).
4. The students are each presented a piece of paper (identical to one another) containing an excerpt from a Dr. Seuss rhyme consisting of 6 to 8 lines of text. Different lines of the text will be one of two colors and contain instances of various types of font styles (e.g. underlines, bold characters, etc.). A colored swatch indicating the student’s designated color will be affixed to the top of the paper to serve as an identification reminder to the student.
5. The test conductors will load the Write activity on the XOs and network the application between the two units for the students.
6. The students will be instructed that they are to duplicate the given paper to the best of their ability using the XO, and that their units are linked to one another so that they will be working on the same document from different machines. In addition, they are only responsible for the lines of text that are the same color as the one they have been designated. They will also be informed that they are free to talk to and help one another however they would like to.

Communication of intent

(To be distributed after the school or institution’s approval of testing, but before the testing date, to all children in the participating class(es). This shall be accompanied by a brief informational/data sheet with details of the OLPC XO Laptop), as well as informed assent and consent forms.)

Dear Parent or Guardian, On Day, Month, Date, 2008, a group of three Rochester Institute of Technology (RIT) students and their class OLPC-project sponsor/external advisor will be visiting your child’s class. As part of their school assignment they intend to briefly observe your child and his/her classmates as they examine and use a new computer laptop designed for school children. The observational session is anticipated to last 15 to 25 minutes and is completely voluntary. Even if you and your child have approved the consent and assent forms provided to you, your child’s participation is not mandatory and your child will not be penalized in any way if he/she does not participate. In addition, your child may choose to withdraw from participation at any time before or during the observations. As part of the study we will be asking your child questions pertaining to his/her:

• access to and experience with computers
• native language and ethnicity
• experience with the laptop computer used in this session.

Your child does not have to answer any questions if you instruct them not to, or if they choose not to do so.

In addition, all the children will be video/audio-taped as they interact with the new computer. We will be taking care not to record your child’s face and our footage will be edited and/or altered if we discover that it contains information directly identifiable to your child. All information collected from this session will be maintained in the strictest confidence and no information will be personally identifiable to your child. Included with this notification you will find more details about the product your child will be using, as well as an Informed Consent and Assent form for you and your child to complete. There can be no participation in these activities if your child does not have both forms signed and approved on the testing day.

Please feel free to contact your child’s teacher or any of us with any questions or concerns.

______________________________________________
Jacob Barber ; jcb9872 at rit.edu ; (phone number)

______________________________________________
Deidra Simone Robinson dsr8347 at rit dot edu ; (phone number)

______________________________________________
Man Truong ; mxt6656 at rit at edu ; (phone number)

Informed Consent

Project Title: “One Laptop per Child” Laptop Computer Use Study

Investigators: Jacob Barber, Deidre Robinson, Man Truong

Purpose: The Rochester Institute of Technology (RIT) would like your child to participate in a study that will help make a laptop computer designed for school-aged children easier to use. Please take some time to review these documents and discuss this study with your family and child. The decision to let your child participate in this study is up to you and your child.

What is involved: Your child will be observed and recorded by video/audio as they interact with and use a laptop computer that was designed for children as a learning tool. We may ask your child to complete a task using this computer and they may be working together with another classmate. By evaluating how your child and other children use this device, we hope to improve on its design to make it even easier for children to use. We will be asking your child several questions regarding his or her age, primary language, ethnicity, and experience with computers before they participate in the evaluation. Additionally, we may ask them some brief questions about their experience working with the laptop computer once they are done. This activity will be carried out at your child’s school during normal class time and will take approximately 15 to 20 minutes. All sessions will be limited to 25 minutes. The evaluations will be carried out by three students from RIT and their project sponsor/advisor. The school superintendent and/or technology coordinator may also be present during the sessions. Since your child will be taken out of class for the evaluation, he or she will not be able to participate in normal school and class activities during this time. However, we have obtained permission and support from the school superintendent, as well as your child’s teacher, to conduct these observational sessions. While we do not foresee any risk to your child during the study, he or she may decide to stop participating at any time. We may also stop your child’s evaluation of the laptop if we judge that it is in their best interest to do so. If we find this necessary, you will be notified and informed as to why.

Confidentiality: These sessions will be recorded with video/audio equipment but your child’s privacy is of the utmost importance to us. Since the focus of our study is on the laptop computer, we will make every attempt to restrict all video recording to the unit itself. Our video footage will be reviewed later and, if we discover any information identifiable to your child (their face), we shall edit or alter the footage so that your child’s identity will remain concealed and protected. Any information regarding your child and his/her participation in this study will be strictly confidential and will not be shared with anyone outside of our group. You and your child’s name will not be used in or linked to this study. The results of the study may be published for scientific purposes but will not disclose your name or include any identifiable references to either of you. Any identifiable information, including these forms, shall be securely saved in a password protected digital format by one member of the testing group.

Stopping: Please understand that participation in this evaluation is completely voluntary. If, at any time, you wish for your child to withdraw from the sesessions, you may do so without penalty.

By signing below, you are indicating that you understand the content of this form and that you approve of your child’s participation in our testing session. You also understand that participation is voluntary and may decline without penalty. Additionally, you understand that you or your child may choose to withdraw at any time, even after a session has already begun, without penalty. I, ___________________, give permission for my child, _________________, (PRINT YOUR NAME) (PRINT CHILD’S NAME) to participate in the above study. __________________________________ ______________________ (SIGN NAME) (DATE) Please sign one copy for your child to return to us and keep the other for your records. If you have any questions, comments, or concerns, please contact either: Jacob Barber @ (phone number) @ (phone number)

Informed Assent

Project Title: “One Laptop per Child” Laptop Computer Use Study

Investigators: Jacob Barber, Deidre Robinson, Man Truong

We are doing a research study about how easy it is to use a special laptop computer made for children. A research study is a way to learn more about people. If you decide that you want to be part of this study, you will be asked to use one of these special laptop computers. We may also ask you to use it with one of your classmates to see how you work together. This will take about 20 –25 minutes during school time. There are some things about this study you should know. We will be asking you some short questions about how you use computers before you use the laptop. We will then have you spend some time using the laptop in different ways. You might use one alone, or we may have you do something on it with a classmate. When you are done, we will ask you a few more questions about how you used the laptop and what you think about it. While you are working with the laptop we will be watching you to see what you do, and also recording you on video so that we can watch what you did at a later time. We will use this information to write a report about what we learned. This report will not include your name or that you were part of the study. We will not share any personal information about you with anyone else. You do not have to be in this study if you don’t want to be. If you change your mind and want to stop after we begin, that’s ok too. You can ask your parents, your teacher, or any one of us (investigators) any questions you may have about it. Your parents and teachers know about this study, too.

If you decide you want to be in this study, please print and sign your name below. I, ________________________________, want to be in this research study. ( PRINT YOUR NAME ) __________________________________ ______________________ ( SIGN YOUR NAME ) ( TODAY’S DATE )

Participant Screener (If a child answers “yes” to the following questions, or cannot complete all the tasks presented, they shall be excluded from testing.)

• Do you have a hard time typing on a keyboard, or does it hurt you to type?
• Do you have any trouble seeing or reading the screen of a computer?
• Can you read the following sentence? (show a separate piece of paper with the following sentence, or another similar sentence, in large, simple font)

From there to here, from here to there, funny things are everywhere.

• Can you tell me what the two following colors are? (the child needs to be able to distinguish between the two colors used in the typing task)

Pre-Testing Survey (This survey is not to be distributed to the child, but instead verbally conducted. The test conductor will record the answers below.)

1. Have you ever used a computer before? Yes No
2. If so, was it a: (show pictures) Desktop Laptop Both
3. Did you use it at: Home School Other
4. Do you know if it was a: (show pictures) Windows Mac (Apple) Other
5. If you have one at home, does it belong to just you or your whole family? Mine Whole Family
6. About how often do you use the computer? Every Day Every Other Day Once a Week Less?
7. How long do you use it for? (ask about parental limits or start/end times)
8. What do you use the computer for?
9. What do you do when you’re on the computer? (circle all that apply)
   • Write
   • Draw
   • Browse/surf internet
   • Instant Messenger
   • Play Games
     • Type? (online vs, Flash games, etc)
     • Other ________________________________

Post-Test Ratings

(This survey is not to be distributed to the child, but instead verbally conducted. The test conductor will record the answers below.)

1. Did you like the XO laptop? Why or why not?
2. Do you like how it looks? Why or why not?
3. What did you like about it?
4. What don’t you like about it?
5. (Scaled Question Below – should be a 5 pt scale as in: Easy / Kinda Easy / OK / Kinda Hard / Hard (or similar for a child’s comprehension comprehension)
6. Do you think it is easy to use?
7. Is there anything else you would like to tell me about how you feel about the XO Laptop or about what you’ve done today?

16 March 2008 Meeting

Draft notes: Image:Meeting1Summary.doc Meeting #1 Description OLPC
Attending: Deidra Robinson, Man Truong, Jacob Barber, Frederick Grose

Goals and Expectations:

Since the OLPC laptops are relatively new, we realize that there is a broad spectrum of areas we could focus on. For purposes of our usability tests we will focus on a piece of software or software related issues. The client realizes that there have been some complaints about the laptop, but leaves it up to us to find the problem area/s. The client would like us to discover the area where we can contribute the most to developers in the time allowed for our testing. In this meeting we did not settle on one particular application or idea; however, we did narrow down possible topics to existing software on the laptops.

I think we are leaning toward testing one of the 'collaboration' facilities after reviewing what's available and how to procede theoretically and practically. --FGrose 21:37, 16 March 2008 (EDT)

It is expected that we will share and track our testing and findings on the OLPC wiki website. Also the client would like to take a very active role throughout this process. There are some plans already in store for the laptops that the client understands we are not aware of and we have come to terms on the limitations of our studies.

Types of users that you will want to represent with participants:

The OLPC laptops are sent to underdeveloped countries for children, teachers, and parents to use as a learning tool. The main audience is children between the ages of 6 and 12, with an even more targeted focus on 6 – 10 year olds. This targeted focus is because the laptops are usually given to the younger population and the students keep them for 4-5 years, so by the time they reach the older ages they will be already fairly familiar with the laptops. Since for many of these children these laptops are this may be the first computer they have ever seen nonetheless use we will aim at finding users with very little computer use. Ideally we would find children who have never saw a computer.

Type of test you are thinking about:

We are looking into types of software testing in addition to doing traditional observation and surveying.

Approximate number of participants needed:

Resources limit the maximum number of participants; to get reliable data we will need, at a minimum, 10 participants.

Our final numbers will also be influenced by what we intend to test and how many units we will have available for testing. For example, testing the collaboration between two users and the XO's "paint" program if we only have access to two units.

Access to the system to be tested:

Frederick Grose (the client) informed us that there are about 6 laptops on campus and is working to get us access to some/all of those machines. Stephen Jacobs, an R.I.T. professor (http://www.it.rit.edu/~sxj/), owns one to three "XO" units and Fred obtained permission to loan one of them to Man Truong on 17 March 2008. The XO was handed off to Man on the evening of 18 March 2008 17:30 (EDT). Since Man had a class from 18:00 to 21:50 and works from 08:00 to 17:00 there was limited time available to examine the unit before the next team meeting. This particular XO will be more thoroughly reviewed by all team members during class on 19 March 2008.

The team also anticipates access to multiple XO units during the OLPC Workshop at R.I.T. on 26 March 2008. Our first experience with the XO's collaborative interface to other physically present XO units may be that evening.

In the meantime, two team members will attempt to installed the SUGAR interface/OS on their own machines with access to software packages and installation instructions from the OLPC wiki. All software is open source so we should have unlimited access. We hope to join these emulations in collaboration with any actual XO units in our possession.

Anticipated special challenges for study:

At this point our biggest foreseen challenge will be finding children with little to no computer experience. Any participants' prior experience with Windows or Mac (or other) operating systems and physical laptop devices will influence their experiences with the XO unit during testing. These effects conflict with the intended user group of children with virtually no exposure to computers and operating systems. This variance will be addressed during the actual study.

NOTE: Heuristic evaluation is not possible this week due to no access to the XO unit.