(Editor's Note: On Monday we posted Oscar Becerra's Letter to the Editors of The Economist in response to the publication's very critical article about Peru's OLPC implementation. Today we're following up with an extensive guest post by Oscar Becerra in which he hopes "to shed light on what seems to be, probably unintentional, misleading information" about the Peruvian project and IADB's study.)

I was surprised by the beginning of the Economist's article "Error Message" (based on the IADB study) that says the Peruvian Una Laptop por Niño project "did not accomplish anything in particular". The IADB study clearly stated that the project "substantially increased use of computers both at school and at home", "positive effects were found in general cognitive skills" and improved "competence in operating laptops in tasks related to core applications (like a word processor) and searching for information on the computer".

The IADB study also says that there was no significant impact on school enrollment or attendance, nor in test scores in Math and Language. Enrollment in target schools was already nearly 100% -- therefore no further increase could be expected. Also, the program focus was not on improving test scores in Math or Language and there were no applications directly aligned with those subjects.

As the person responsible for Una Laptop por Niño for almost five years I hope to shed light on what seems to be, probably unintentional, misleading information.

Background

The IADB study, which I personally supported and helped to deploy, did not include important background information regarding the Peruvian basic education environment:

• A January 2007 census evaluation of 180,000 Peruvian basic education teachers showed 92% of them lacked basic Math reasoning skills and 62% of them did not read at 6th grade level, 27% were at levels of zero or below.
• After 200 hours of remedial education run by local universities' faculties of Education, 13% of the teachers were still at the same zero or below level.
• The program designed by the government to improve the average quality of teachers at public schools will require at least 10 years to complete.

The above described situation left us with the difficult choice of waiting 10 years to do something or begin in parallel. A 2007 study by McKinsey found that the world's most improved school systems had in common their concern for teacher quality and getting the best people into teaching - a finding that really reduced our options. We also knew that ICT skills had been identified as key ingredients for success by many organizations (see, for example, this report). We believed that putting 21st century tools into the hands of children would be a good way to begin working on the problem while a better teacher workforce was being developed.

It was never the program's primary goal to improve Math and Language test scores. What we expected was that the children's lives would be improved by giving them more options for their future -- something the study has proven because the children are more proficient in the use of 21st century tools (Table 5 of the study shows a dramatic increase in both access and use of computers by the treatment group) and have better cognitive skills (Table 7). It remains to be seen if with appropriate adjustments in strategy those improved cognitive skills will translate into better test scores. I don't agree with the Economist's biased reading of the study.

In spite of the fact the program did not aim directly to improve test scores we agreed with the IADB's proposal to study impact on test scores because if positive impacts were found we would have gained 10 years. The fact that there was no impact on test scores should not be a surprise nor a reason to dismiss the program. It means that we need to adjust the strategy and reinforce teacher training as well as trying to capitalize on the improved cognitive skills of children in order to pursue more ambitious objectives.

The IADB Study

I would like to comment on specific areas of the study and present alternative interpretations to most of what has been circulating recently. I don't think any of the material circulating is unbiased and this is also my personal opinion.

"The program did not affect the quality of instruction in class" (p.3)

I cannot agree with the statement that activities done "might have little effect on educational outcomes (word processing, calculator, games, music...)". Being able to work with a word processor and a calculator should be seen as an educational outcome in itself. The paragraph implies that Math and Language scores are the only educational outcome to be expected.

The study goes on to mention that "on the positive side, the results indicate some benefits on cognitive skills". What needs to be done, and we tried to prepare the project to be able to do so, is to build on those improved cognitive skills by integrating additional material as teachers' quality improves. For example, we were able to integrate external stakeholders into our effort: the National University of Engineering has an application development laboratory that is working on the design of educational games specifically aimed to improve Math scores; a local firm developed a reading comprehension application that has proven effects and runs on the XO's, offering a possibility to improve children's reading comprehension scores from sixth to eleventh grade.

"limited information about how to integrate the computers provided into regular pedagogical practices" (p.6)

This is true but the problem arises when we identify "regular practices" with correct or desirable practices. It is worth noting some teachers still consider hitting students with a stick a regular practice. In a school system with a teacher profile like the one described it is not reasonable to expect regular practices are something we want to reinforce. Choosing to work only with teachers with good practices is not an option as it leaves many teachers out of the picture - it was not a decision to ignore them. Many good teachers have made the extra effort, based on the 40 hours training, and achieved very good results, but the good teachers are a small minority. The main issue here is the lengthy and difficult process of teacher assimilation of technology. I remember a meeting of Peruvian education officers with Clotilde Fonseca in 1998 when she was president of Omar Dengo Foundation in Costa Rica. Fonseca said that after 10 years of the Programa de Informatica Educativa, Costa Rican teachers were able to talk about Piaget correctly. Not that they were able to apply it properly to their classroom practice, but at least they were moving in the right direction.

"lack of Internet access and the fact that the laptops did not run Windows..." (p.7)

We knew from the beginning that universal Internet access would be lacking. This led us to design what we called "the portable Internet", a 2GB USB drive with educational pre-selected pages allowing children and teachers to experience the feeling of navigation and to include a reduced 30,000 entries version of Wikipedia in Spanish. That was what we called "asynchronous connectivity" meaning that the pages would be periodically updated based on teachers and student requests. Regarding Windows, with support from OLPC and Microsoft we ran a test project with Windows based XO's and had to deal with the high requirements in terms of Memory - Storage and virus infection problems. The project was generally successful, however, Microsoft decided to abandon it because they were not developing their platform for the XO. We were convinced that the ICT skills would be easily transferable to other platforms in the future since upgrading to new versions of Windows can be as complex as switching from Sugar to Windows. One unintended advantage of asynchronous connectivity was the protection against access to inappropriate content that we couldn't expect from teachers who knew less than their students about ICT.

"The central outcomes of the study are achievement and cognitive tests" (p.10)

The authors cannot be more specific about what they found and the whole document focuses on the fact that there was no measurable impact in achievement as measured by Math and Language test scores AND that there were positive effects in cognitive skills as measured by Raven's Progressive Matrices test. Why so-called independent analysts decide to write about one and ignore the other may be a matter of study in itself.

"Motivation toward attendance and homework" (p.11)

The study says that its findings contradict what has been suggested regarding motivation (p.2). Since I was the one who suggested this, I must say that in a study we made, we measured intrinsic motivation towards school work, not towards attendance or homework. Therefore, the study does not prove that the suggestion was wrong. The variables measured that showed improvement were: interest - pleasure in school work, relation between effort and results obtained, perceived competence for school work, creative stress, perceived selection of what to do, and improved personal relations. The only variable that did not improve was perceived importance of school work, which was very high from the beginning.

"Computer Access, Use and Skills" (pp. 13-15)

The study describes very promising results in this section. For example, only 13% of laptops malfunctioned at some point and half of them were successfully repaired. This is a demonstration of the XO's ruggedness and the success of our training program, which devoted 8 hours to technical maintenance by teachers. Also promising was that the theft ratio was only 0.3%. It's worth noting that we decided to give the computers but not the property because in those places where parents were mistakenly told the computers were theirs to own, many of them immediately decided to try to sell them for cash.

One negative finding of the study is that the students used the computers mostly during school time. Given that there were no specific Math or Language activities we must assume that they were taking time away from regular school work and using it for OLPC activities. The fact that the test results show no impact may be interpreted as "the teachers work in class has no impact on test scores" -- something I am afraid is not far from the truth.

"self-perceived school competence ... evidence of small negative effects" (p.16)

I don't agree with the implication of these negative effects as a decrease in self-esteem. What may be inferred is that children realize that they are not well prepared for school, a finding that is good if it motivates them to work harder to achieve what they want. What I found is that children in non-OLPC schools have extremely high perceived competence for school work, in spite of their dismal results, something the presence of the XO seemed to be correcting for good.

"Academic Achievement and Cognitive Skills" (pp. 16-17)

I find this section very instructive. The study states that there is no pedagogical model linking software with particular curriculum objectives. That is true because we were convinced that, in order for curriculum-related software to work, we needed well-educated, competent and well-trained teachers. Since we could only provide the training component there was no way to ensure this approach would work. The fact most teachers did not find the training enough is a proof training only was not enough. Re-educating them was well out of our reach.

It is rewarding to learn that the approach chosen did not replicate reported negative effects of home computer usage on grades. Another positive impact neglected by most readers.

The main positive effect described here is the access and use of computers translated into improvements of general cognitive skills. In my opinion, this means that the foundation is in place on which to build and that the task is to continue working to achieve the desired results. It is to be seen if the new government administration will be willing to keep building or will decide to forget about it and begin again. It is promising the person in charge (Sandro Marcone) was the first director of Proyecto Huascarán, a predecesor of the General Directorate of Educational Technologies.

One surprising positive result is the 4.6 to 6 months advantage in cognitive skills progression for the treatment group, which means a 30 to 40% improvement in just 15 months. Why the analysts chose to ignore this remains to be understood.

"Discussion" (pp. 19-20)

I don't share the conclusion "governments should consider alternative uses of public funds". It is known that improving teachers' quality is a long-term effort (Korea took 40 years to improve its education system). Something needs to be done apart from waiting. Whether building on top of improved cognitive skills will result in an improvement in test scores is something that might be worth pursuing. It seems that the current administration has a strong focus on teacher training, which will surely have a positive effect.

The mention about improved IQ in countries as emphasized by some researchers is in line with a study by Nina Hansen that suggests improved IQ test results in Ethiopian children who participated in an OLPC program.

In conclusion, the IADB study is valuable and will hopefully serve as a guide to strategists in getting the most out of the investment made. In no way does it support the Economists's "Error Message" article or the many interested parties who are trying to use the study to push their own agendas. I will have to finish this as I did at the beginning of the project by quoting Miguel de Cervantes' "Don Quijote" when he said "Let the dogs bark Sancho, it's a sign that we keep advancing".

Oscar Becerra was the Chief Educational Technologies Officer in the Peruvian Ministry of Education between September 2007 and July 2011.