OLPC research/Evaluations review 2009-2010

Evaluations of OLPC projects: A review of 2009-2010 reports

This review was presented at the Innovation in Evaluation workshop on April 4th and 5th, 2011 at OLPC Cambridge, MA.

We conducted a review of the evaluations to projects that implement the One Laptop per Child’s 1:1 educational computing model across the region. For each evaluation report we describe the indicators measured and tools used for data collection. The indicators are divided into two groups: social impacts and impacts related to learning. Some reports also measure aspects related to program implementation. We also include a summary of the key findings from each report. Finally we summarize the most common indicators measured throughout the eight reports as well as most common tools used for data collection. The second part of the document includes two overview reports, followed by a summary of the main conclusions obtained from the evaluations analyzed.

Educational Monitoring and Evaluation of Plan Ceibal (Uruguay)

Report date: 2009 Country: Uruguay Author: Plan Ceibal Evaluation Department http://wiki.laptop.org/go/OLPC_research#Official_overviews.2C_December_2009

Plan Ceibal was launched in 2008 with the vision of bridging the digital divide as a factor of inequity in the country, through universal access to computers and Internet. In 2009, the distribution of 395,000 XO was completed to children in public primary schools in the country.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Locations for internet Access Frequency of use of XO by children	Operating status of XO Teachers’ opinion on training Evaluation of Plan Ceibal Problems at school level	Activities preferred by children Activities used for homework Time needed to learn to use the XO (children) Uses of XO Motivation to use the XO in class Teachers: Instructional planning	Self-administered surveys for children, teachers, principals and parents Computer based text production by children Individual and group interviews

Key Findings:

• The internet sites have changed from cyber cafes (65% in 2006 to 25% in 2008 in the country side), to school (32% in 2006 to 67% in 2008).
• The teachers believe that most children take less than two weeks to learn to handle the XO, and most learned by individual exploration.
• Over 50% of teachers report they include the XO in instructional planning at least once a week, and up to 21% almost daily.
• 92% of children reported using the XO for homework.
• Browse activity is the mostly preferred activity of children of all ages, a preference that increases with grade. The same pattern occurs with Etoys, and Record activities.
• Activities such as Write, Paint, and Memorize tend to be less preferred by children as they advance in grades.
• About 90% of teachers have used Browse and Write activities mostly for teaching purposes. Browse is mostly used in higher grades and Write activity in lower grades. Half of teachers at schools without connectivity opt to use the writing activity, and secondly -1 in 5 - the browser to search for content on the XO digital library. Finally, 1 in 10 used to chat and recording of photos and videos as alternatives for use in school.
• 77% of children said they like to work more with the XO in the classroom.
• Access to the XO not only closes the gap in access to computers and the Internet but also its use levels the students who don’t have a computer at home with the ones who do.

OLPC pre-pilot Evaluation Report (Haiti)

Report Date: 2009 Country: Haiti Author: Inter-American Development Bank http://wiki.laptop.org/go/OLPC_research#OLPC_Haiti_pre-pilot_evaluation_report_.28IDB.29

Haiti received more than 10,000 XOs part of the initiative Give One Get One program; the pre-pilot program was conducted during May and July 2008, implemented during a summer camp with 116 students.

Indicators related to Learning Impacts	Tools
Uses of the XO, activities, and frequency of use Teachers: Teacher’s knowledge of the XO Characteristics of teaching environment: Level of student attention and focus Teacher discipline system Level of participation and encouragement of the teacher Relationship between teachers and students Student attitudes and behaviors	Classroom observations of teachers and students Tracking the use of the XO through the journal Interviews to students, teachers, administrators Case Studies

Key Findings:

• The distribution of XOs was 1:2, 1 laptop shared by a pair of students. Only a limited group of 4th graders took the XO home throughout the duration of the camp.
• The attention span of students decreased because of the following factors: when they shared the XO between students, short duration of battery power, low commitment / motivation from the teacher.
• 4 of 17 activities available on the XO (Write, Write, Browse and Painting) represented 88% of the use of the XO from camp participants.
• The tendency to use certain activities were strongly related to the instruction provided by the teacher, the use of these activities showed a steady increase, while the exploratory use of other activities decreased.
• 40% of students explored to reach a point of frustration or confusion, and were shy or reluctant to seek help from the teacher
• Students and teachers use the XO to practice or improve reading and writing in Haitian Creole and French, 56% of teaching staff noticed an improvement in the students in this aspect as well as the positive effects of having content in these languages for improvement in reading comprehension.
• 40% of students believe the XO could facilitate learning and provide opportunities for the future.
• Teachers think that the XO could boost school attendance, interest and independent thinking, thus contributing to a better understanding and learning

Synthesis of Overall assessment report of One Laptop Per Child (Paraguay)

Report Date: 2010 Country: Paraguay Author: Alda Foundation

4.000 XOs were delivered in 10 primary schools in the community of Caacupé in 2009. The goal of the program Paraguay Educa is to promote an educational system that uses ICT to foster meaningful, collaborative, and student-centered learning.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Attitude of principals and teachers about incorporating technology Internet Access Communication tools with the XO	School management Infrastructure and connectivity Laptop repairs	ICT Skills Frequency use of XO Using Internet Peer to peer Learning Motivation to use XO in the classroom Activities used	Interviews Questionnaires Polls Discussion groups Observation Analysis of tasks and documents

Key Findings:

• There is a positive incidence of incorporation of the XO in schools.
• 70% of teachers stated that access and use of ICT contributes to satisfaction on a professional and personal level.
• The computer serves as a means of communication for children with their families abroad.
• Factors that have a positive influence in the development of project: acceptance and pedagogical perspective by directors, the presence of the community around the project, shared expectations for directors and families.
• The motivation and interest children show when using the XO facilitates and optimizes learning situations.
• Teachers expressed a positive attitude toward using the Internet and to the contribution of ICT as a resource for learning.
• There is good connectivity and access to internet
• 96% of children demonstrate knowledge and/or skills on ICT from having access to the XO
• The XO enables the exchange and interaction among peers

Experimental evaluation of OLPC in Peru

Report Date: 2010 Country: Peru Author: Inter-American Development Bank

Peru has more than 300,000 XO distributed in more than four thousand schools. It is currently the largest OLPC project in the world. The goal of the program Una Laptop por Nino is to improve the quality of public primary education especially for children in the most remote areas of extreme poverty.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Expectations and attitudes of family and teachers Expectations and perceptions of children Frequency of use of the XO out of the classroom	Laptop ownership Teacher Training Internet access and electricity	Cognitive Skills: Mathematics, communication, ICT skills Other skills: Problem solving, collaboration, teamwork, self-guided learning. Dropout and attendance Teachers: Pedagogical practices	National tests (communication and mathematics) Interviews Questionnaires to families, teachers, directors Classroom observations

Key Findings:

• Teachers and parents show greater enthusiasm and higher expectations in schools that are part of the program.
• There is a low percentage of students who can take home their XO
• 95.2% of schools have electricity, and 1.4% of schools have Internet access.
• There is a demand for more training and technical/pedagogical support by teachers
• There were no significant differences in national test results obtained by students in the treatment group compared with the control group.
• Students with XO are more critical about their education, their school and their own academic performance.
• There is a positive correlation between test results of ICT use and the reading comprehension test

Evaluation report on OLPC (Colombia)

Report Date: 2010 Country: Colombia Author: Ana María Velásquez, PhD

The Merani Foundation and Pies Descalzos Foundation implemented a program to improve skills in Spanish and Mathematics through the use of the XO in three schools in different cities in Colombia.

Program Implementation	Indicators related to Learning Impacts	Tools
Community Awareness Teacher Training Tech Support Internet connection	Mathematics and Spanish skills	Questionnaires adapted from math and Spanish sections of SABER exams (national exams for 3rd grade)

Key Findings:

• Students from 2 out of 3 schools intervened during the 18 month period, presented substantial higher performance than students without intervention or that had just been exposed to the program for 6 months.
• The third school didn’t show significant academic advantages based on the test results as a result of being in the program. The implementation of such school was in charge of a different institution than the other two.
• The participating schools are also performing two other improvement programs simultaneously with the XO program.