OLPC Oceania/Evaluation

Oceania Community - News - Content - Evaluation - Languages - Countries
Cook Islands -French Polynesia - Kiribati - Nauru - New Caledonia - Niue - Palau - Papua New Guinea - Samoa- Solomon Islands - Tonga - Tokelau - Tuvalu - Vanuatu

Evaluation

Solomon Islands has published the report of an independent evaluation by ACER 
Which can be accessed here

The main purpose of the projects is to evaluate the suitability of the OLPC Program to meet the Education Improvement plans of the Pacific Islands. The evaluation should be suitable for

a) The local teachers, principles and parents

b) The Ministry of Education

c) Regional Funders

In the initial projects we will conduct evaluation in partnership with the Ministry of Education and other stakeholders

We will extract what is workable and introduce a simple evaluation and feedback component from the start. This wiki page is part of that feedback process.

We will partner with experts in the field to help in the Evaluation

Each country will develop local interpretations of this top-level strategy.

Please note that USP has been funded to conduct an in depth evaluation of the OLPC Ocenia initiative with particular focus on Niue Details will be published on this site.

OLPC Oceania is also in discussion with the Australian Council for Educational Research (ACER), particularly in relation to the Solomons trail, but also for a more wider understanding of the impacts of OLPC. They have published a preliminary report called Evaluation of OLPC programs globally: a literature review 03/09 with the intention of identifying existing approaches to the evaluation of the OLPC programs globally. This document will be updated at regular intervals.

Introduction

Measurement and Evaluation is a key element in any education improvement initiative and particularly with Laptops. An important issue to decide at the beginning of the project is just what are you trying to measure. For example, if we measure how well students learn to use the computer or like the laptop, we may miss many other key benefits.

There have been many studies conducted on ICTs in education with many different results, both positive and negative.

Many of these studies are quite old, dating from around 2000-2004 and the technology and approach to using ICTs in Education has changed quite a lot, so be careful about people quoting research and studies on ICT in Education.

One other key issue to understand is that many of these studies are based on a "Computer Lab" model and not the OLPC approach of every student owning a laptop. Computer labs are usually used for students to learn about computers, not for student to use the computer to learn

The approach to M&E for OLPC may be very different from previous studies. We suggest you think carefully about waht things you are expecting OLPC to change and then develop ways of measuring this The Solomon's Ministry of Education have drawn up an Evaluation Framework for their OLPC project which may be useful. Please see the following section for information on this.

NOTE that experts in education evaluation agree that you cannot develop a meaningful frame work to describe or measure the direct impact of ICT in student learning. It is best to look for impacts of the use of ICTs on students, learning environments, teachers and pedagogy, schools systems and policy and practice.

M & E is a specialist skill and we probably don't have the resources to conduct full studies on the outcomes from OLPC, however, we should evaluate the project in our terms and use the evaluation as a learning exercise.

A couple of things to note about evaluation

• It is a continuous exercise, not just a once off thing at the end of the project

• Evaluation and reflection is core to any learning exercise.

• Experts suggest you start with one or two key indicators and as a group, discuss them, gather data over a period (minimum of one term) and evaluate the results. As you become more familiar and confident in using evaluation tools, add other indicators.

If you are interested in a stronger research based approach to evaluating the impact of OLPCs, the Ethnographic for ICT approach developed by the Queensland Institute of Technology and supported by UNESCO, could be a good approach

The ethnographic action research approach for the research and development of ICT projects is based on combining two research approaches: ethnography and action research. Ethnography is a research approach that has traditionally been used to understand different cultures. Action research is used to bring about new activities through new understandings of situations. We use ethnography to guide the research process and we use action research to link the research back to the project’s plans and activities.

This may be a good approach as Education in the Pacific is focusing more on traditional knowledge and culture and there have been many issues raised about the access to mostly Western Culture on the Internet

For more information on this approach, please go to the UNESCO Portal

http://portal.unesco.org/ci/en/ev.php-URL_ID=13276&URL_DO=DO_TOPIC&URL_SECTION=201.html

A Copy of the Ethnographic Action Research (EAR) is available at

http://www.twine.com/_b/download/11w8fyrcz-w5/b0bsl79jx59rwtw7lr978psshbdk7tsxlb1w4wlpmfhprmt/11w8fyrcz-w5/b0bsl79jx59rwtw7lr978psshbdk7tsxlb1w4wlpmfhprmt/ear_handbook.pdf

This handbook is well written and simple to understand and gives a practical way to go about evaluating an OLPC project and helps to develop plans for doing things better.

Resource people needed

Evaluation should include as many stakeholders as possible. Examples include

• Teachers,
• education officers,
• independent observers,
• champions,
• community leaders and parents
• volunteers
• ... and don't forget the children!

In the preparatory consultations, identify and select one teacher and one community representative and train them to collect monitoring data. Also identify an education officer and an independent person (maybe from an NGO with an interest, prepared to resource the process) who will be able to visit the school at key intervals – maybe after 2, 4 and 8 weeks.

Helpful information

InfoDev, part of the World bank and specialising in ICT for development, have a good range of information about ICT in Education

Please refer to the following web page

http://www.infodev.org/en/TopicPublications.4.html

In particular, they have a number of resources on Measurement and Evaluation for ICT projects in Education Their Quick Guide can be seen at

http://www.infodev.org/en/Publication.150.html

Note that in section one of this document, there is a very handy section on "Do's and Don'ts in Monitoring and Evaluation"

Other recommended references include the series of research papers on the impact of ICT on learning and teaching published by the Department of Education and Training, Government of Western Australia. (Dr. C. Paul Newhouse, 2002)

http://www.det.wa.edu.au/education/cmis/eval/pd/reading/reading1.htm

Please note that this report and other reports should be read with the folowing in mind

1) Some are quite old, dating back to 2000 and things like OLPC, social networking and web 2.0 and many other developments since then may mean that what they are talking about has changed. For example, the above report talks about Computer Literacy, not Information Literacy, I guess because Information literacy was not developed in 2000. Another example is Wiki Educator as a resource that was not even considered in 2000

2) Most reports are written from a context where teachers are well trained and schools have access to good teaching and learning resources. This may not be the case in your school. Your needs and environment are quite different from a western school.

3) Many reports are written in a context of western learning models and we know that these are not working all that well in indigenous cultures. One example of this is that OLPC facilitates collaborative learning which is more aligned with traditional learning. Another example may be how ICTs (and in particular OLPC) supports traditional knowledge and culture. Such areas may not be considered in many reports.

We have written some notes pulling the above together, which needs developing. The diagram below illustrates.

Click here for notes on the potential impacts (draft).

Results

SPC will publish any M&E results relating to OLPC around the world on this page and we encourage you to place your plans and results here as well. If you are unsure about editing wiki sites, there is a short tutorial available on this site, or SPC will post any material you send to us.

Early indications and feedback from Oceania trials

There are various reports in each country page on this wiki - browse to each using the menu top of page. However, here are some examples:

• Feedback from trials sites in PNG, Nov 2008
• from http://www.youtube.com/watch?v=2oVFKcjOVB8
  • 00:55:00 Kids from school in Oceania linked up with NZ School with Kids in Melbourne and Sri Lanka and started to exchange info and teach each other stuff.
  • 00:59:00 Ian THOMSON testifies on the impact he noticed and teachers notice. The teachers all of a sudden finding out another approach to teaching: learning to learn and them seeing how much better it works with the kids to get to grips with things they have to learn.

Framework

An example of an evaluation framework from the Solomon Islands trials is shown here

A framework of indicators could include:

Appropriation

• Children
  • How often are they using the laptops, using which activities, on their own and with others
  • What skills can they demonstrate, individually and in groups (measured non-obtrusively) Measurement: Hold a session at school where the children are free to use the laptops on their own or in groups, as they wish, and observe the group dynamics and note which activities are being used. Observe how groups are interacting – for instance, are all the participants involved in the learning and sharing? Allow them to explain what they are doing in an informal setting, without asking specific questions. Some problem solving tasks can be set. The session can be recorded with digital camera, video, and a survey form or template report for the observer to use.
• Teachers
  • Are they learning to use the laptops in their class-work, and if so how, how much, what ideas have they developed regarding integrating the laptops into their teaching. Measurement: (see “educational impacts” below)
• Families
  • What are the parent’s attitudes to the children using the laptops
  • Are other family members using the laptopsMeasurement: Hold a parents evening and hold open discussion. Follow up with simple questionnaire that the parents can take home. The community rep can follow up and help each family complete the forms. Committee can arrange for observers to visit homes, with permission of the families.
• Community
  • What are the community’s attitudes to the laptops
  • Is the community finding the laptops useful, not only in education?
  • Is there any change in the way the community regards the schoolMeasurement: Community meeting

Educational impact

• Have the teachers adapted to more discovery learning / constructivist methods
• Have the laptops helped the teachers directly in their teaching (i.e. access to resources, collaboration / communication, subject knowledge, pedagogical knowledge especially regarding constructionist learning....)
• Is there a change to more child-centred approach in the classroomMeasurement: With the teachers’ permission and preferably led by the teacher, observe lessons in practice, non-obtrusively. Follow up with interviews using prepared questionnaires. The teachers can also keep log books of how they have used, the laptops, what ideas they have and other feedback.

Information literacy

• How many children, teachers and families are accessing information on the Internet, and are communicating via email (compared with before)Measurement: Can be included in the questionnaire (see “Appropriation: families”) plus observers can record their observations. Also via tasks set for the group session of children, as described above.

Laptops/technical

• Technical problems with the laptops and how they were dealt with
• Charging issues
• Connectivity issues
• Keyboard issues
• Language issues (some feedback for the localisation program)

Evaluation by teachers and district officials

At Dreikikir (see PNG page) the teachers came up with their own suggestions.

The following approaches were decided by teachers, and agreed with the education officials present:

• Teachers would keep a log book / diary and update it with any feedback on how the laptops are being used, new ideas on classroom integration as they develop, and feedback on student’s uses. Also feedback of problems and issues.
• Every morning, the first lesson is an “oral session” where teachers can discuss the laptops with the children. This session will be used to get daily feedback.
• Parents evenings and staff meetings will also be used to get feedback and share ideas.
• The district school standards officer and education advisor have been trained and fully participated in the deployment. This is very important, as they are available “on the ground” to make evaluations and carry reports from the school to the Province and Department.
• A volunteer to be based at Dreikikir for 2 weeks will provide additional evaluations on behalf of OLPC Oceania. Guidance on this from SPC will be crucial.

Things to look for in an OLPC project

The following is a collection of important things that OLPC projects have noted.

This is not intended to be a comprehensive list, or an evaluation template, however it may give you ideas of things to evaluate during your project.

Education Approaches of an OLPC Project

A core element of the OLPC program, as opposed to the tool of the laptop, is “Learning learning”.

The OLPC program is based on the Constructionalist Education theory, or learning by doing. It is also has a strong collaboration learning approach, encouraging students to share their experiences and learn together.

Some Education ministries are reforming their system to a more student centric approach to replace the teacher centric or instructional approach.

Generally there is a strong belief that the education system needs to change and with today’s affordable technology, especially the OLPC Laptop and Internet, the tools are available. The status quo for education in developed and developing countries is not acceptable.

Several OLPC projects are driven by the need to reform the education system. For example, Peru had tried many times to make incremental changes to their under performing system with no success. Today, they find the Laptop and Internet are transforming the system very effectively.

The degree to which Learning by Doing is being incorporated in the classroom might be the basis of an evaluation framework. For instance, in the Solomons, Learning by Doing is a theme that is being introduced into the primary school curriculum. Such an indicator would be directly linked to policy, as well as to the design characteristics of the XO laptop.

It is up to each country to design such a framework for evaluation, according to their own priorities and policies.

List of possible outcomes

Here are some things that have been mentioned as positive outcomes from an OLPC project. Many of these observations come very early in the project.

This list could also be considered reasons for launching a project.

1. Reduced absenteeism
2. Greater engagement by children
3. Empowerment of teachers
4. Increased esteem of teachers
5. Reduced need of discipline
6. Less disruption by students
7. Increase in self esteem by students
8. Increased “connection” with the world (other cultures, languages, ideas)
9. Increased involvement by parents
10. School becomes the centre of the community
11. Development of a sharing culture (child to child, school to community, teacher to parent)
12. Much better results than with Computer Labs