OLPC research/Evaluations review 2009-2010

Evaluations of OLPC projects: A review of 2009-2010 reports

This review was presented at the Innovation in Evaluation workshop on April 4th and 5th, 2011 at OLPC Cambridge, MA.

We conducted a review of the evaluations to projects that implement the One Laptop per Child’s 1:1 educational computing model across the region. For each evaluation report we describe the indicators measured and tools used for data collection. The indicators are divided into two groups: social impacts and impacts related to learning. Some reports also measure aspects related to program implementation. We also include a summary of the key findings from each report. Finally we summarize the most common indicators measured throughout the eight reports as well as most common tools used for data collection. The second part of the document includes two overview reports, followed by a summary of the main conclusions obtained from the evaluations analyzed.

Educational Monitoring and Evaluation of Plan Ceibal (Uruguay)

Report date: 2009 Country: Uruguay Author: Plan Ceibal Evaluation Department http://wiki.laptop.org/go/OLPC_research#Official_overviews.2C_December_2009

Plan Ceibal was launched in 2008 with the vision of bridging the digital divide as a factor of inequity in the country, through universal access to computers and Internet. In 2009, the distribution of 395,000 XO was completed to children in public primary schools in the country.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Locations for internet Access Frequency of use of XO by children	Operating status of XO Teachers’ opinion on training Evaluation of Plan Ceibal Problems at school level	Activities preferred by children Activities used for homework Time needed to learn to use the XO (children) Uses of XO Motivation to use the XO in class Teachers: Instructional planning	Self-administered surveys for children, teachers, principals and parents Computer based text production by children Individual and group interviews

Key Findings:

• The internet sites have changed from cyber cafes (65% in 2006 to 25% in 2008 in the country side), to school (32% in 2006 to 67% in 2008).
• The teachers believe that most children take less than two weeks to learn to handle the XO, and most learned by individual exploration.
• Over 50% of teachers report they include the XO in instructional planning at least once a week, and up to 21% almost daily.
• 92% of children reported using the XO for homework.
• Browse activity is the mostly preferred activity of children of all ages, a preference that increases with grade. The same pattern occurs with Etoys, and Record activities.
• Activities such as Write, Paint, and Memorize tend to be less preferred by children as they advance in grades.
• About 90% of teachers have used Browse and Write activities mostly for teaching purposes. Browse is mostly used in higher grades and Write activity in lower grades. Half of teachers at schools without connectivity opt to use the writing activity, and secondly -1 in 5 - the browser to search for content on the XO digital library. Finally, 1 in 10 used to chat and recording of photos and videos as alternatives for use in school.
• 77% of children said they like to work more with the XO in the classroom.
• Access to the XO not only closes the gap in access to computers and the Internet but also its use levels the students who don’t have a computer at home with the ones who do.

OLPC pre-pilot Evaluation Report (Haiti)

Report Date: 2009 Country: Haiti Author: Inter-American Development Bank http://wiki.laptop.org/go/OLPC_research#OLPC_Haiti_pre-pilot_evaluation_report_.28IDB.29

Haiti received more than 10,000 XOs part of the initiative Give One Get One program; the pre-pilot program was conducted during May and July 2008, implemented during a summer camp with 116 students.

Indicators related to Learning Impacts	Tools
Uses of the XO, activities, and frequency of use Teachers: Teacher’s knowledge of the XO Characteristics of teaching environment: Level of student attention and focus Teacher discipline system Level of participation and encouragement of the teacher Relationship between teachers and students Student attitudes and behaviors	Classroom observations of teachers and students Tracking the use of the XO through the journal Interviews to students, teachers, administrators Case Studies

Key Findings:

• The distribution of XOs was 1:2, 1 laptop shared by a pair of students. Only a limited group of 4th graders took the XO home throughout the duration of the camp.
• The attention span of students decreased because of the following factors: when they shared the XO between students, short duration of battery power, low commitment / motivation from the teacher.
• 4 of 17 activities available on the XO (Write, Write, Browse and Painting) represented 88% of the use of the XO from camp participants.
• The tendency to use certain activities were strongly related to the instruction provided by the teacher, the use of these activities showed a steady increase, while the exploratory use of other activities decreased.
• 40% of students explored to reach a point of frustration or confusion, and were shy or reluctant to seek help from the teacher
• Students and teachers use the XO to practice or improve reading and writing in Haitian Creole and French, 56% of teaching staff noticed an improvement in the students in this aspect as well as the positive effects of having content in these languages for improvement in reading comprehension.
• 40% of students believe the XO could facilitate learning and provide opportunities for the future.
• Teachers think that the XO could boost school attendance, interest and independent thinking, thus contributing to a better understanding and learning

Synthesis of Overall assessment report of One Laptop Per Child (Paraguay)

Report Date: 2010 Country: Paraguay Author: Alda Foundation

4.000 XOs were delivered in 10 primary schools in the community of Caacupé in 2009. The goal of the program Paraguay Educa is to promote an educational system that uses ICT to foster meaningful, collaborative, and student-centered learning.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Attitude of principals and teachers about incorporating technology Internet Access Communication tools with the XO	School management Infrastructure and connectivity Laptop repairs	ICT Skills Frequency use of XO Using Internet Peer to peer Learning Motivation to use XO in the classroom Activities used	Interviews Questionnaires Polls Discussion groups Observation Analysis of tasks and documents

Key Findings:

• There is a positive incidence of incorporation of the XO in schools.
• 70% of teachers stated that access and use of ICT contributes to satisfaction on a professional and personal level.
• The computer serves as a means of communication for children with their families abroad.
• Factors that have a positive influence in the development of project: acceptance and pedagogical perspective by directors, the presence of the community around the project, shared expectations for directors and families.
• The motivation and interest children show when using the XO facilitates and optimizes learning situations.
• Teachers expressed a positive attitude toward using the Internet and to the contribution of ICT as a resource for learning.
• There is good connectivity and access to internet
• 96% of children demonstrate knowledge and/or skills on ICT from having access to the XO
• The XO enables the exchange and interaction among peers

Experimental evaluation of OLPC in Peru

Report Date: 2010 Country: Peru Author: Inter-American Development Bank

Peru has more than 300,000 XO distributed in more than four thousand schools. It is currently the largest OLPC project in the world. The goal of the program Una Laptop por Nino is to improve the quality of public primary education especially for children in the most remote areas of extreme poverty.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Expectations and attitudes of family and teachers Expectations and perceptions of children Frequency of use of the XO out of the classroom	Laptop ownership Teacher Training Internet access and electricity	Cognitive Skills: Mathematics, communication, ICT skills Other skills: Problem solving, collaboration, teamwork, self-guided learning. Dropout and attendance Teachers: Pedagogical practices	National tests (communication and mathematics) Interviews Questionnaires to families, teachers, directors Classroom observations

Key Findings:

• Teachers and parents show greater enthusiasm and higher expectations in schools that are part of the program.
• There is a low percentage of students who can take home their XO
• 95.2% of schools have electricity, and 1.4% of schools have Internet access.
• There is a demand for more training and technical/pedagogical support by teachers
• There were no significant differences in national test results obtained by students in the treatment group compared with the control group.
• Students with XO are more critical about their education, their school and their own academic performance.
• There is a positive correlation between test results of ICT use and the reading comprehension test

Evaluation report on OLPC (Colombia)

Report Date: 2010 Country: Colombia Author: Ana María Velásquez, PhD

The Merani Foundation and Pies Descalzos Foundation implemented a program to improve skills in Spanish and Mathematics through the use of the XO in three schools in different cities in Colombia.

Program Implementation	Indicators related to Learning Impacts	Tools
Community Awareness Teacher Training Tech Support Internet connection	Mathematics and Spanish skills	Questionnaires adapted from math and Spanish sections of SABER exams (national exams for 3rd grade)

Key Findings:

• Students from 2 out of 3 schools intervened during the 18 month period, presented substantial higher performance than students without intervention or that had just been exposed to the program for 6 months.
• The third school didn’t show significant academic advantages based on the test results as a result of being in the program. The implementation of such school was in charge of a different institution than the other two.
• The participating schools are also performing two other improvement programs simultaneously with the XO program.

Evaluation of OLPC pilot project in Solomon Islands

Report Date: 2010 Country: Solomon Islands Author: ACER (Australian Council for Educational Research)

The report refers to a pilot project at 3 schools in Solomon Islands in July 2008.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Changes in school registration and attendance Community participation	Children’s opinion of the XO Program awareness Opinion of teachers and principals about program objectives and impacts Technical difficulties of the XO	Uses of the XO Learning new things Independent Learning Peer collaboration Motivation towards learning Teachers: Changes in pedagogical practices Instructional planning Incorporation of XO in teaching practices	Interviews to students, teachers, directors, parents, and other members of community Questionnaires to officials from Ministry of Education.

Key Findings:

• Children in grades 3-6 report they use a greater variety of activities and emphasized the use of the XO especially for learning.
• Children in grades 1-2 use the computers to learn to read, write words and learn mathematics. The activities they use at school include drawing, painting and others.
• The frequency in which they use the XO depends on the availability of charging stations for the laptop.
• The study found the following positive impacts on students: self-learning, sharing and collaboration among students, and greater motivation
• The benefits identified by parents include: increased and improved learning, especially in vocabulary, reading and writing, improving spelling and pronunciation, they learn techniques to photograph and record.
• Other areas of benefit discussed by parents include socialization, school attendance of children sharing with family and children's pride of ownership.

Innovative Learning in Ethiopia

Report Date: 2010 Country: Ethiopia Author: OLPC Ethiopia in conjunction with Eduvision and the University of Groningen

5.000 XOs were donated to four schools as an initiative of the Give One, Get One Program from OLPC. The ECBP (Engineering Capacity Building Programme) implements the program. Their objectives are to transform the teaching-learning process from repetitive to construction based. This report reflects the period from September 2008 to May 2009.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Psychological, social, and cultural changes: Self concept measurements : self esteem and self efficacy Social networks Cultural values Educational aspirations Communication tools with the XO User perceptions of the XO	General process of implementation	Academic performance Type of uses of the XO in and out of class	At the school: Class observations Focus groups with teachers and with students Half-yearly Test scores (pre and post) Teachers’ and students’ journals Out of school: Interviews to parents and other members of the community Process samples (extracting data from laptop) Plans to measure social impact:Longitudinal monitoring Regular interview to students and teachers.

Key Findings:

• Two months after receiving the XO 79.1% of children suggest that they use it to learn. 17.6% only use it to play and 9.8% for communication purposes.
• Children who own an XO typically get a 3% higher score in their half-yearly tests when compared to the control group.
• In classes where teachers make extensive use of the XO there is increased academic performance of up to 13%.
• 41.5% of children use the XO with their parents, 34.9% with their friends at school and 22.6% with their friends outside school.
• Children with XO value money, tradition and religion to a lesser extent compared with the control group.
• Children in rural areas showed greater motivation to attend school after they obtained an XO.

Introducing XO Laptops in four Nigerian schools. First year Evaluation (SEED-OLPC)

Report Date: 2010 Country: Nigeria Author: Gerstein and Associates, Licensed Clinical Social Workers

Four schools in Nigeria (2 primary, 2 secondary) are part of the OLPC-SEED (Schlumberger Excellency in Educational Development) project, which was launched in the summer of 2009. The mission of the program is to contribute with educational opportunities in the communities where Schlumberger people live and work and to engage youth in science and technology.

Indicators related to Social Impacts	Program Implementation	Indicators related to Learning Impacts	Tools
Students’ aspirations Teachers’ attitudes Effects over schools and communities	Quality and functionality of the XO	Technological skills Frequency use of the XO Attitude towards learning Ways children learn to use the XO Teachers: Ways teachers use the XO Activities used by teachers on the XO	Logic Model: Short term impact Surveys Interviews Self reports from students and teachers

Key Findings:

• About two-thirds of high school students reported an interest in becoming an engineer of some kind.
• 73% of high school students has become more studious and attentive to their studies and improved their overall learning.
• Groups reported interest in having more access to internet
• Both students and teachers reported using the XO for educational purposes.
• Most elementary students indicated they use the XO once or twice a week.
• 63% of high school students use the XO at least once a day and 96% said they take it home every day.
• Students and teachers reported to have learned the same activities on the XO.
• Students are using their XO for homework and research.

Summary of Common Indicators

Social Impacts	Impacts related to Learning	Program Implementation
Opinion/attitude of teachers and principals regarding the XO	Integration of teaching tasks with the XO	Technical difficulties/ XO repairs
Opinion/attitude of children regarding the XO	Motivation to use XO by children/ Attitude towards learning	Opinion of the program
Internet Access	ICT skills	Understanding of the software and internet access
Access to technology	Activities used in class or for homework	General implementation of the program
Opinion/attitude of families regarding the XO	Peer learning and collaboration
Creation/modification of Social networks	Type of use of ICT on XO
Frequency of use of XO during leisure time	Frequency of use of XO in school
Type of use XO during leisure time	Academic performance
Vocational aspirations of students	Time to learn how to use the XO
Communication Tools	Learning to use the XO
	Independent and self-guided learning

Summary of Common Measurement Tools

• Interviews, Questionnaires
• Observations, Focus Groups
• Case Studies
• School tests, standardized tests
• XO Journal

Part 2: Overview Reports of OLPC projects

We reviewed two overview reports of OLPC projects followed by a summary of the main conclusions obtained from the evaluations analyzed. These reports analyze various projects of OLPC in different countries:

• Evaluation of OLPC programs globally: A literature review

2010 by Dita Nugroho and Michele Lonsdale ACER (Australian Council for Educational Research) http://wiki.laptop.org/images/a/a5/OLPC_Lit_Review_v4_Aug2010.pdf

• Evaluations in OLPC: what for? What has been done, what could be done?

2010 by Pierre Varly File:Evaluation in olpc pierre varly.pdf

Conclusions from Overview reports

• Assessment models, as well as the implementation of the projects themselves, vary greatly in part by the operation of the entities involved in the initiation and implementation of projects.
• The evaluation results are affected by differences in the implementation of the project.
• The results of existing evaluations tend to be positive, highlighting the impact of education and attitude in students, the effects on teacher-student relations and their impact on the community
• OLPC deployments clearly reduce the digital divide and the majority of children use XO as their first technology equipment.
• The expected results of OLPC projects include improved self-esteem and motivation, higher attendance and better learning outcomes. However, only some are focused on performance as measured by scores on academic tests.
• Most of the results report better motivation and attitude and the reduction of repetition rates. However, they don’t inform how the project has contributed to improved learning.
• The focus of evaluations has been formative evaluation: what has been done? How does the community perceive the project? What students are doing with the XO?