In a public report with limited distribution dated July 10 2009, Inspectors G.Aramburu and Y.Delgado of the Initial and Primary Education Council of the Public Education Administration of Uruguay present what I believe is the first ever published and so far only instance of large-scale objective research of an indicator of XO use in education.
Examining impact on XO children
- See the full report here: Circular 10 Técnica July 10 2009
- See the project lineup here: Circular 6 Técnica May 8 2009
This most valuable piece of research regarding the impact of the OLPC concept almost risked to be passed over and forgotten. While at OLPC News we go to considerable effort to bring you up to date information, somehow we missed this, and despite its importance and being almost a year old by now, these documents have not yet been mentioned in any research data list I am aware of.
What Is Special About This Report
- It is objective. So far, be it by design so as to push a makeshift ideology or agenda, be it by ignorance of proper research design techniques and procedures, I know of no research on XO use that consistently follows a straightforward clear method based on objective evidence. Authors of other reports seem to have attended Research 101 in the "if it feels good, it's methodologically sound" School of Parascientific Research. This one report tells things as they are - a very refreshing thing.
- The sampling was wide. 3rd and 6th grades in the largest, medium and smallest schools in every "distrito" in Uruguay (26 or 102? Being addressed to "those in the know", the report doesn't spell out what might be common knowledge there).
- The data got published. Even though addressed to an original audience of a little over a hundred people, thanks to the procedures of the CEIP it is available worldwide. This is even more extraordinary in that the results are not quite what was desired in terms of the "image" of Ceibal. It is SOP that most data at odds with the "feel good" school is gently suppressed or delayed. The authors of this report dare to actually use the result of their research to re-design their work and policies. What a new concept!
- The report was timely. It took only two months, from the research being announced to publication, pretty good anywhere
So, What Did They Find?
Looking at the Class Plan logs, out of 9 days from April 27 to May 8 2009, 3rd grade classes teachers fixed to use the XOs for some class activity 5 times, while 6th grade ones did so 6.4 times, on average. At about 5 class activities per day, that makes a total for these 9 days of 45 learning activities, thus XO use was planned for about 13% of the grand total of class activities.
Of these, 1.4 and 1.7 times is the average planned for 3rd and 6th grade for curriculum-related use besides Internet-supported curriculum use, this later being 1.7 and 2.4 respectively. Both grades averaged 0.8 times planned to learn to use the XO. 1.1 and 1.5 respectively times appears as scheduled internet use in class possibly without connection with curricular activities.
From the conclusions:
"Analysis of the data show a scant use of the XO in developing the educational proposal by the teachers. The average for each type of activity did not show differences that can be considered significant.
Despite the low average for activities to learn use of the XO, it appears that in 6th grade the highest average is related to the use of Internet search for information with precise intervention of the teacher with the objective of contributing to the construction of knowledge."
An Opinion
Congratulations big time to Mss. Aramburú and Delgado. They demonstrate that honest, well designed objective data is the best tool to be able to redefine how we use XOs in class to foster success in learning.
With all due respect, I actually find 5 and 6.5 activity times out of 9 days to be pretty good, in any case better than what I expected. I do agree that more could and should be done, say twice a day at least, and I further congratulate them that their report not only closes with fresh points to ponder, it also includes an extensive collection of ideas that are already in use by Uruguayan teachers and Districts, shared there to encourage emulation of better use of the XOs in class.
A caveat might be in order as to the validity of the data in respect to real life use of the XOs. The teacher planning log does not mean things actually get done - many of the "best" and highest qualified teachers I knew when I did time as their colleague in the Uruguayan educational system were impeccable at filling their planning logs with what ought to be there, like Music, and PE and such, activities they never actually did, but certainly were in the log for any evaluator to see.
Maybe these inspectors, well aware of their 'hood, know that 5 and 6.5 appearances in the logs actually really means "escaso uso" (limited, scant usage), because reality is often different. In any case this does not invalidate the research, which in terms of certainty and validity is heaps of light years ahead of anything else that has come from Ceibal ever, or from anybody who uses surveys of opinion in lieu of objective data.
Yamarant on Research Methods
The best way to ascertain XO actual use would be to look at Journals, or in the case where stuff gets backed up to the server, simply datamine that, something that apparently has not yet been done. So called concerns for privacy just require some decency and honesty by the researchers, no big deal and SOP of real research. It is actually easier to design such datamining to be totally anonymous than many alternatives.
What gets my peeve is that so far Scientific Research and its Methods seem very much a failed assignment in ICT4E Monitoring and Evaluation. It is like, being seen as a Social Science, pedagogy finds in wishes and hubris what it then calls the facts to advance its latest fashion. It is as if the foundations of science were not enough for OLPC initiatives; as if procedural methods basic to academia were not right for this MIT-initiated, Boston-based quest that counts more PhDs as organizers or supporters than anything the like ever seen before.
The amazing thing is that normal and even extraordinary people accept and often justify this state of things, in the name of serving kids... Without apparently noticing how the loss of credibility is hurting these initiatives both short and long term.
When evaluating evaluations, something I do a lot since I am looking to learn all I can, I seek objective evidence. Openness. Honesty. Skill in separating the grain, what is, from the chaff, what we wish it were. This study is a good beginning, shared with you none too early, and hopefully, not to late.
Great article - too bad that the research wasn't published in english. Glad to see that objective research exists!
I always find very interesting articles but in English, this time the situation is reversed. May be you want to use google translate like me.
Thank you, we are working on the translation of course, it's good stuff, well thought out, though we wish we could have something closer to the raw data. If it's any consolation and while you wait, the Spanish translation of this article is ready :-)
Traducción en castellano de este artículo está en http://educalibre.cl/?p=112
I am a strong supporter of olpc, but we really need a whole lot of objective research. Some say this is not needed because olpc is promoting abilities that standard educational tests don't cover. That is an exaggeration, as standard skills like reading are part of the goal. But if there are unconventional goals, like learning to learn, then research methods should be developed for testing them, too.
Unfortunately I can not read Spanish but I can read numbers. And I do not see any...
Can you tell us where the "large-scale objective research" statement is based on?
Is there more data than the single rudimentary table of the 3 page report?
Is there any follow up? (this is almost a year old by now)
Is it available in other formats (for easy machine-based translations)?
As I mentioned above, I really would have wanted more raw data. I will be happy to respond all your questions tomorrow, as right now I have to deal with another issue, sorry, so do please put those questions, but I'll need a bit before I get back to you
Hi Yama
I agree it is a good intention of a report, but I wouldnt consider it large-scale in any way. 3 schools X 19 departments X 2 classes each X 20 students (?) is about 2000 students sample, out of 420000 machines deployed not a really significant number for any analysis. The report itself is not very clear on the procedure to analyze data, or how to make it objective. I believe that after almost 3 years Uruguay MUST have much better reports than this, but they are certainly not sharing them, in spanish, english, or swahili. For example, what happened to the online evaluation that took place this year? That should be a good source of large-scale objective data, but we havent heard back from them. Uruguay is a very good worlwide example but unfortunately a very reserved one. For me that isnt good news
regards
I agree with the concerns about Uruguay.
They do not even share their software modifications or the support schemes they developed.
Rumor has it they are trying to sell them (!) to other deployments.
But actual performance data/evaluations it may be still early. We still do not have concrete data from Main that is an older, almost equally large and presumably more "capable" deployment of 1-to-1 computing.
IF Ceibal is reselling their software modifications and support schemes then... good on them!
At least then they would be sharing what they've done and giving us guidance on the real efforts and resources involved. It would be the first such instance where we have real, credible analysis on what a full 1:1 deployment costs.
The research from Uruguay, albeit one year old, does present a reality that is all too common with laptop deployments. In our lengthy experience with the integration of laptops into classrooms, the simple provision of laptops into a classroom will have little impact on student learning or enhancing the teaching environment. Also, I am struck at how we focus on the issue of research and isolating the impact of technology in the classroom. Credible researchers in the field of education have repeatedly told me that the Human Factor in the classroom plays a formidable role and isolating one specific tool, approach or teaching methodology is almost impossible. The OLPC folks should realize that solutions do exist elsewhere, sharing of information and supporting each other, in Uruguay, Canada, Peru, etc. will go much further than getting involved in discussions that are very circular.
Bottom line: Providing ICT to classrooms is a no brainer. Supporting/training the teachers and the students in the effective and transformational usages of ICT is the key. Research is important but let's take the time to see the impact. In our case, it took over 4 years to see and measure any real effect, so any measurements of the OLPC effect are very premature.
The ICT gap that now exists between classrooms and the "outside world" is becoming bigger and bigger, regardless of where we live in this world.
The research from Uruguay, albeit one year old, does present a reality that is all too common with laptop deployments. In our lengthy experience with the integration of laptops into classrooms, the simple provision of laptops into a classroom will have little impact on student learning or enhancing the teaching environment. Also, I am struck at how we focus on the issue of research and isolating the impact of technology in the classroom. Credible researchers in the field of education have repeatedly told me that the Human Factor in the classroom plays a formidable role and isolating one specific tool, approach or teaching methodology is almost impossible. The OLPC folks should realize that solutions do exist elsewhere, sharing of information and supporting each other, in Uruguay, Canada, Peru, etc. will go much further than getting involved in discussions that are very circular.
Bottom line: Providing ICT to classrooms is a no brainer. Supporting/training the teachers and the students in the effective and transformational usages of ICT is the key. Research is important but let's take the time to see the impact. In our case, it took over 4 years to see and measure any real effect, so any measurements of the OLPC effect are very premature.
The ICT gap that now exists between classrooms and the "outside world" is becoming bigger and bigger, regardless of where we live in this world.
"They do not even share their software modifications or the support schemes they developed."
It's pure smoke and mirrors. Nothing is happening in Uruguay. The laptops were distributed and not much else has happened. It is not that they don't want to share; they just don't have anything of interest to share.
"Rumor has it they are trying to sell them (!) to other deployments."
There are no buyers (they are the only large-scale deployement other than Peru) and there is nothing to sell. So, there is very little to worry about on this front.
@mavrothal:
"They do not even share their software modifications or the support schemes they developed."
Inexact: see for example the list of activities developed in Uruguay ( http://drupal.ceibaljam.org/?q=lista_descargas ), see the next (the 7th!) developers gathering aka CeibalJam ( http://drupal.ceibaljam.org/?q=node/758 ), see what other developers are doing & announcing: SupervampireNinjaZero (http://www.supervampireninja.com/ ), CeibalTerminal ( http://sites.google.com/site/sugaractivities/ceibalterminal ), El Barrio en Juego ( http://elbarrioenjuego.com/ ), Memoria Animal & Rompacabezas Hidatico ( http://www.ceibal.edu.uy/contenidos/juegos/memoria_animal/index.html ), etc.
Conclusion: don't be unfair, be aware of what great things the "South" is dreaming & doing collectively!
funny how one of the very few "activities" listed by Sami is a little olod-fashioned Flash game of rote memorization!
I was given to the Ceibal Project as a "correction" of a previous gift that apparently was too expensive and didn't provide any good results.
http://www.mgap.gub.uy/PRENSA/Juegos%20Did%C3%A1cticos%20para%20Plan%20CEIBAL.pdf
The developers' site is here:
http://www.smart-branding.com/proyectos/memoriaanimal.php
@me: Are these ones too old schools for you?
* FotoAventura is another activity/game developed by smart-branding, with the support of the uruguayan Ministerio de Turismo y Deporte:
http://www.smart-branding.com/fotoaventura/pc/index.html
* The customization with uruguayan sounds and rhythms of the activities TamTam Jam and TamTam Edit (see: http://www.ceibal.edu.uy/index.php?option=com_content&view=article&id=926:tamtam-y-los-sonidos-uruguayos&catid=86:bajar-archivos-download&Itemid=230 )
Sumy yoy either cann't or don't want to understand or you simply think we don't.
I'm talking "software modifications or the support schemes" and you are pointing to a couple of activities most of them beta.
Where are the Uruguay OS images and most important, the OS _sources code_ posted Samy???
Isn't part of the GPL2/3 that XO/Fedora/Sugar is distributed under to post them? (unless you rewrote everything from scratch...)
Where is the support software source code posted and the XS-mods?
This response of yours, if it is official to any extend, just confirms the "rumors"
@mavrothal
Disclaimer: I am not related in any way with the Plan Ceibal or any uruguayan institution. I am an individual contributor working with many others, in particular in relation with OLPC France, an independent NGO and as such, I observe and share my observations (either positive or negative) with you and others individuals. You should ask the right people for your - admitedly right - concerns.
@mavrothal:
"Where are the Uruguay OS images and most important, the OS _sources code_ posted "
Im am not going to do your homework, however, since you were asking so kindly...
Here's the download & instructions page for the - supposedly - last uruguayan image: http://www.ceibal.edu.uy/index.php?option=com_content&view=article&id=1015:la-imagen-uy-802&catid=86:bajar-archivos-download&Itemid=230
and:
http://www.ceibal.edu.uy/index.php?option=com_content&view=article&id=707:icomo-flasheo-mi-xo&catid=86:bajar-archivos-download&Itemid=230
Diclaimer #2: I am nor a developper nor a layer, so please don't ask me too complex questions related to code or licensing issues!
Excellent....
The 6-days valid image (the product) is available!
Now, can you also dig out any source code site to see how all this is happening?
This would convince anyone that the rumors are just rumors and indeed Plan Ceibal is using and distributing FREE code
I am posting a request in the Sur List about this, hope we get that sorted out.
http://lists.laptop.org/pipermail/olpc-sur/
It has been my experience that even though OLPC / Sugar was all about kids being able to modify Activities, etc, sources tend to be actually hard to find, as there doesn't seem to be anything like a central repository, but then, I am no longer involved in actual programming, though I wish I were
There is for OLPC
http://dev.laptop.org/git/
and SugarLabs
http://git.sugarlabs.org/
OK, guys, fine, the report is bad, but it's the very best so far in terms of objective data and validity, so let us cut it some slack until you guys show us something better.
Yes, it would be great to have raw data.
Yes, it would be wonderful the data covered all of the country, instead of just 3 schools per "distrito". But that's what "sampling" is all about. Well done samples can get as little as a few hundred responses, and make reasonably good guesses from that for a whole country. I believe the sampling here was rather on the side of impeccable. I have no reason to assume the reality is that much different from the rest, the numbers do feel rather where other indicators point after a few thousand Sur emails.
BTW I believe there are more distritos than Departamentos, I have a sense there might 102 of them, I get that number for how many "Inspector Distrital" seats were assigned a couple months ago.
Yes, there is indeed a report from a massive Ceibal study, but it is survey-perception based, which is an excellent method, to measure perceptions but not much else, and which shows, of course, much higher perceived "levels of use", which accurately points out what is that people do want to make themselves believe.
People want to believe in Ceibal. We all do. I mean, even when things don't really go right, we can learn from it and keep moving.
This was what Aramburú & Co were after. They took the chance, they dared to face reality, report on it, and build on it.
Others have tried also, and maybe I was too unkind on them, though, I stand in that this one is very different from anything I've seen before.
My dad is a geologist. His trade has been to go out in the middle of nowhere and get a few chunks of rock and come back and figure out what it all means. I guess that I am taking after him in figuring out this is a gem of a report, rough, certainly, small, yes. But it can lead us the way of the mother lode, which in my view has a lot to do with good objective methodology.
Translation of Circular 6, which was the one setting up the study (give me some time to complete the other circular. Thanks be to Google Trans, which gave me the first draft that I then mangled further)
National Public Education Administration
Primary Education Council
Technical Inspection
Montevideo, May 4, 2009
Circular No 6
National, Department and Area Inspectors,
"The Plan Ceibal is a big bet Uruguay is making for equity, democratization of knowledge and educational improvement. " [footnote says UNESCO, Ceibal in the XXI century society, P SB. 11)]
The challenging task of optimizing the various possibilities that computers offer for building knowledge and improving communication is up to the educators. It competes the supervisors the guidance and monitoring of its use in classrooms and homes. A clear overview of the situation will allow to intervene as needed at the institutional and classroom levels. [veeery fancy word for "classroom level" is used here]
The Elementary Education Division promotes research around the integration of XO in the educational context, research that commits to generate instances of reflection in all teachers from different levels of intervention.
For this reason, we ask that all Area Inspectors of Primary Education carefully analyse planning logs developed by the Teachers identifying activities that integrate the use of the XO.
For the sample election we will work with the "A" groups in of 3rd and 6th grades among three urban schools when sorted ascending [by size?] in each district. The first, third and the last will form that sample. In the departments of Canelones and Montevideo research
will be conducted in all schools which started the pilot in the year 2008.
The study will be limited to the activities planned by the teachers for the period from 27 April to May 8 this year. We present a table for recording the data. In the same differentiate by
type of activity that includes the use of the XO. It is not the same to plan an activity for children to learn their use, as it is for work that involves School Program content (eg, classification of triangles). It also is not the same to ask children to use computers merely for Internet access, or to ask to locate specific information on the Internet with a specific intent for a classroom subject involving to analyse the information collected to promote construction of knowledge about specific program content.
[unclear if "construir" below means "construct" or "build", no differentiation in Spanish]
(Table 1)
Quantity of Activities Planned by the Teacher for the purpose listed
study from April 27 to May 8
- knowledge of XO use
- internet search without precise teacher intervention to contribute to construction of knowledge
- internet search with precise teacher intervention to contribute to construction of knowledge
- teaching and application of knowledge related to program content
(space for each school / class data)
(table 2) Days in which XO use was planned during the period from 27 April to 8 May.
27/4 28/4 29/4 30/4 4/5 5/5 6/5 7/5 8/5
3rd A
3rd A
3rd A
6th A
6th A
6th A
By email [it is sent :-)] [you will receive] a spreadsheet to integrate data from these two tables above from all districts in the jurisdiction in a single electronic document to enable process data at regional and national levels.
The processed data at the level of each jurisdiction will be sent in that spreadsheet
to the General Inspectors by email by May 31 also attaching an analysis and interpretation of the information collected (one page) and the action areas that are planned from that state of affairs.
Some questions that can guide the eyes of supervisors in these instances of visiting schools.
• How are the XO used to improve the production of written texts (elimination of repeated words, use of synonyms, incorporating adjectives and adverbs, differentiation of sentences, use of punctuation, capitalization, connectors, organization of the text) using the applications (cut, paste, move, insert, use of dictionary, etc..)?
• What computer programs are used in teaching activities that teachers plan in the various subjects? Is it a powerful resource for the promotion of learning according to the contents addressed?
• How is work with the computer within educational proposals encouraged :
at home?
working in teams?
integrating children of different grades on projects involving
sequence of activities, following processes, use of different resources
(Photographs, graphics, synthesis of information gathered from internet searches, etc..?
• Which teachers do develop systematic work with computers, promoting the construction of learning about different curricular content? Have there been instances of exchange with other teachers to witness this work?
• What instances are promoted at the classroom and institutional level for work involving both the students and family members?
Some of the difficulties that arise in institutions for work with the XO are not ignored
but valuable work can be done notwithstanding these constraints (lack of access to internet in some areas or simultaneously, computers that need repair, etc.).. Work in teams and offline offers valuable opportunities to work in all disciplines. Supervisors are key actors who can contribute to opportunities becoming strengths to optimize the educational possibilities offered by this valuable investment made by the country and has been materialized through the joint efforts of different agencies, institutions and individuals.
Graciela Arámburu
Technical Inspector
I think we have to consider the research in terms of the methodology and objectives that it had. I would classify it as rather descriptive research about what is in use, who is using, for how long etc. It does show that they are in use, and clearly the government also achieved part of it's objective to provide more equal access.
That's quite different to comparing test results before/after (and not just standardized tests, but tests of problem solving ability, comprehension, etc) between a control case (preferably with the same amount spent on a conventional intervention) and an OLPC equipped school.
Given a marked difference in the results before/after one could then use such research to argue for causality.
Also one has to consider the context of Latin America; actually Uruguay's education system before the introduction of Ceibal / OLPC was pretty good (didn't dig into all the background but things like time on task, provision of feedback, access to libraries, class sizes, etc) were all in reasonable shape.
In Africa / S. Asia class sizes are often 50+, time on task (total num of hours in school per year) is less than half the average in developed countries, structured feedback can be almost non-existent... Hence research conducted in Uruguay does not generalize to validate or invalidate what would happen in such a fundamentally different context.
Unfortunately our attempt in Afghanistan didn't come out quite as comprehensive as we would have liked owing to resource constraints... But that will change as per the briefing note that we published.