The ready-to-test idea

=D

Assessment

An important aspect of education is the provision of assessments and examinations, both so that people can find out how much they have learned and also as an indication to oth package. If one sets up a system to broadcast learning texts and learning software to a specific location or set of locations then the cost of supplying the information to one student or to many students is the same.

When, however, the matter of assessments and examinations arises, there is no free economy of scale as for adding additional viewers of broadcasts, for each assessment or examination needs the attention of a human assessor on an individual basis. Twice as many assessments may not need twice as much assessor time, as some parts of an assessment may take place in an examination setting where, within limits, a few extra students need no more invigilators: however, any individual aspects such as time spent in marking examinations rises as the number of students rises, maybe then not quite linearly but an increase nonetheless.

The issue of qualifications for distance education that is provided upon the OLPC platform is potentially a big issue. Any qualifications must be properly invigilated and assessed and that raises issues of financial cost and access.

One solution would be simply to say that distance education provided upon the OLPC platform is not for qualifications, thereby avoiding issues of provenance of such qualifications and of access to examinations by end users.

However, people learning by distance education using the OLPC system may feel that they would like to have qualifications, or some recognition of their learning.

A partial solution

I have therefore tried to devise an infrastructural technique that does not purport to provide qualifications yet does provide useful feedback to end users and to third parties whom end users may choose to contact, such as potential employers and colleges, yet which technique has the potential to ease the route towards qualifications and may hopefully act as a catalyst to more qualification gaining opportunities becoming available.

I am calling this infrastructural technique "ready-to-test". The idea is essentially that standardized tests may be made available as content on an OLPC laptop and that an end user may attempt such tests. There is not a return link to anywhere. A person taking the test may, if he or she passes the test in private, declare that he or she is "ready to test" on that particular test under examination conditions. The name ready-to-test implies that no qualification has been received, just that someone considers that he or she is ready to take the test.

How might this be useful?

A person could seek employment and write to the employer that he or she is ready to test on a list of specific tests at interview, or an employer could specify in a newspaper advertisement for a job that candidates should be ready to test on specific tests. The tests would be readily available upon the OLPC platform. The intention is that the tests would normally accompany a learning package that teaches the knowledge and skills that are tested.

Such tests would be intellectual property and intellectual property rights in them would exist.

Possible uses of the tests.

This section needs to be rationalized as regards what will be the licensing of the tests. If they are under the Creative Commons licence used for thsi wiki, then charging fees could not be done. However, the tests could be a potential source of revenue for the OLPC project if licensed to businesses for staff recruitment purposes: though they would always be free for the children. This is a topic which needs the opinions of members of the community and of OLPC management please.

Use in the home;

Use if an employer tests, as part of a job interview process, a person who has written in speculatively saying that he or she is ready to test on one or more specific tests;

Use under a fee-paying licence for an employer who advertises vacancies where the employer's requirement is for job applicants to be ready to test for specific tests in the available range;

No use allowed by an employer to test existing staff;

Use where a person in a position of responsibility at no charge writes a letter, either a specifically addressed letter or a "to whom it may concern" letter, stating that an individual has been observed satisfactorily completing the test by the writer of the letter;

Use under a fee-paying licence where an educational institution uses the tests to test students on an institutional basis, the level of fee, which could potentially be zero in some circumstances, being decided upon the circumstances, such as whether fees are charged to students and whether the educational institution is a profit oriented organization or a non-profit organization;

Other use under a fee-paying licence, the level of fee, which could potentially be zero in some circumstances, being decided upon the circumstances.

Format of the tests

Tests could be in a variety of formats.

Some tests might have only one question and just display a question in response to a push of one button on the hand held infra-red control device and the answer to the question in response to a push of another button on the hand held infra-red control device; the student writing the answer down on a piece of paper after pushing the first button and before pushing the second button, possibly after having to carry out a calculation by hand, possibly taking several minutes to evaluate the answer.

For example, a test to multiply together two whole numbers, each of four digits. For example, a test to multiply two quaternions together. In each of these examples, the specific items to be multiplied would be produced using random numbers by the test program and the test program would compute the correct answer ready for display when needed. So observation of someone producing the correct result in each of three successive attempts at the test would be good evidence of competence at carrying out that skill.

Some tests could require the student to enter data into the laptop, perhaps to use some program that is supplied as part of the OLPC project using some specific data posed by the test in order that the student may be tested on his or her competence to use that data and get a result, that result then being checked for correctness by the computer.

Other tests might have many questions, in order to test knowledge of some particular topic, such as, for example, a topic from history. These tests could possibly be multi-choice answer type questions where the computer keeps score or they could have questions where the student is asked to write down answers before pushing the enter key so as to reveal the correct answer and move to the next question. By revealing the correct answer after the student has made the attempt at answering, the tests would also have a learning aspect, so that if the student did not get the correct answer to a factual question then hopefully he or she would learn by then seeing the correct answer. Certainly the tests would only be applicable to specific factual knowledge, detailed reasoning in an answer would not, as far as I am aware, be testable in such a computer based system - though one does not know what level of artificial intelligence programming some programmer might bring to bear on the problem at some time in the future.

These ready-to-test tests could also be made available on the world wide web when the ready-to-test tests are being broadcast in countries where world wide web access is available, even if only businesses tend to have world wide web access in those countries.

The ready-to-test system could possibly be of great value if a number of standardized tests were available throughout the world. This would mean that a person who was able to get the correct answer to the test in one location could make a claim as to being ready to test on that test anywhere in the world and the nature of what were being claimed would be immediately recognized.

Certainly, converting a claim of being ready to test into a qualification or into some evidence of ability acceptable to someone else is a big issue. However, consider that a school teacher were able to observe a student complete a test satisfactorily and were then willing to write a letter on official stationery that the student had been observed satisfactorily taking that test. Such a letter would have a value not as a qualification as such but as a reference that refers to widely known standardized tests. For example, such a letter might, say, state that the writer of the letter had observed the named student satisfactorily complete the test "RTT-M004 multiplication of two whole numbers each of four digits" at a certain date, time and place and state that details of the RTT-M004 test could be obtained from a particular place on the internet or as hard copy from a particular source.

There is also the possibility that community colleges could offer a facility where someone who claims to be ready to test on some collection of tests may attend at the community college and take a number of tests during, say, a half hour period and receive a certificate from the community college of having passed the tests, if he or she demonstrates that he or she can do so. Thus the student could have learned at home by distance education and achieved a skill level of being ready to test at home and then receive a qualification by attending at a test centre on one occasion. In such circumstances the test centre might perhaps be some distance from the student's home, where attendance on a regular basis to attend classes might not be possible, so that the combination of distance education for learning and the ready-to-test technique in order to get ready to test, both using a laptop at home, and then examination for a qualification on the same tests at a local assessment centre could be a useful infrastructural route for providing a complete learning, testing and qualification system.

Certainly, the ready-to-test concept might be more useful for some topics than for others, both due to the nature of the topic being tested in the sense as to whether that particular skill or knowledge can be satisfactorily tested using the ready-to-test technique; and also due to the level of interest in various topics being tested using the ready-to-test technique. For example, topics covered in widely available examinations for qualifications widely used by colleges and training organizations and widely specified by employers might not be particularly of interest as topics for ready-to-test tests, simply because of the widespread use of those widely available qualifications and examinations: however, maybe people who cannot get to a training course might nonetheless appreciate a ready-to-test test in one or more of those topics.

However, more specialized topics, where specific qualifications are not easily available might be very suitable for the ready-to-test approach. For example, a collection of tests on quaternion arithmetic, the use of quaternions for encoding positions and rotations in three-dimensional computer models, taking the logarithm of a quaternion and the exponential of a quaternion in order to smooth combined rotations, and so on might be a very useful specialist mathematical collection of tests, useful for a student to test his or her progress in learning about quaternions and also useful to an organization wishing to employ staff to work on three-dimensional computer models to test job applicants at interview.

In this document I have referred to examples based upon multiplication of four digit numbers and the multiplication of quaternions, deliberately so as to indicate the wide range of skill levels of topics for which the ready-to-test system could be used to provide tests for students learning by distance education provided upon the OLPC platform.

I have also mentioned the possibility of the ready-to-test system being used to test knowledge of historical topics.

Provision of tests in specific topics will require quality input from qualified professionals who can provide good quality questions that are of relevance. Input from content authors for the OLPC platform is also needed in order to make questions that are produced available upon the OLPC platform.

The ready-to-test concept has a potential for use to accompany distance education applications upon the OLPC platform. Hopefully this potential will be realized by practical application of the ready-to-test technique. The big potential of the ready-to-test technique is that if the tests are standardized throughout the world then a person claiming to be ready to test on a particular test is making a claim to a level of competence such that what is being claimed is specifically known throughout the world and that the test to check out that claim can be taken wherever the tests are available on an OLPC laptop or on the world wide web.

Note about how this started

In the Software Ideas - Education page there is a section entitled The "ready-to-test" concept with a link to a document in an external webspace.

Yet webspaces can come and go, so the content of the referenced external web page was copied into this wiki page and an edited version for the OLPC project was produced by the author of that document so that it could then be edited by other people, so it is alright to do so.

Hopefully a system which could be useful for the One Laptop per Child project can be produced.

The original version of the document is in the first version of this page accessible in the page history should anyone choose to read it.

Correlating Ready To Test with Human-Marked Exams by sampling

To increase the value of Ready To Test marks to an employer, without the expense of an exam-marker for each extra child, sampling could be used.

A fixed sample of say 100 students nationwide could sit both the Ready To Test assessment and the equivalent Marked-Exam and be used to show how well the computer-based Ready To Test results (0-100%) correlate with human-marked exam-results (0-100%). The results could be presented as a two-dimensional X-Y scatter-chart :-

Fig 1 - Exam Results versus Ready To Test results

The information would also be shown as a table with average exam-mark and variance for each 10% ready-to-test result band :-

Ready-to-test  Exam Mark Average  Variance  Children in Band
0-10%          5.1%               1.2%      10
10-20%         14.9%              2.3%      9
etc.

Some subjects such as mathematics may correlate well and the two sets of results would cluster tightly around the '1 to 1 correspondance' diagonal line. Other subjects, which are difficult to assess manually and automatically, such as History, may only have a loose correlation. Individual results may lie quite some distance from the diagonal, but overall, there may still be enough of a correlation to show to employers for any child in the country that good marks on Ready To Test usually mean that a child would have done well in an equivalent human-marked exam.

In history for example, good/poor results on automatically-marked multiple-choice tests for key dates, facts, terminology and alternative explanations of an event should show some correlation at least with good/poor performance in an exam where essays test their grasp of a subject.

A particular school-leaver would have a standard report for each subject, that they could show to an employer. It would contain the scatter graph with their mark, such as 87%, indicated by a vertical line (see Fig 1). This would show the various exam-marks that children in the sample obtained who also got 87% in the Ready To test assessment and whether the variance was wide or narrow. The report would also have the information for that school-leaver as figures :-

Ready-to-test Mark  Exam Mark Average  Variance  Children in 80%-90% Band
87%                 88%                3.4%      11

To an employer, a good Ready To Test result doesn't PROVE that someone would have obtained a certain qualification and be a good candidate for a job, but it increases the likelyhood. It's like a good reference.

--Ricardo 16:27, 12 August 2007 (EDT)