Some obvious notes on Mitra and Crawley (2014)

Sugata Mitra and Emma Crawley have published a paper on the SOLE experiments based in Newcastle, UK titled Effectiveness of Self-Organised Learning by Children: Gateshead Experiments.

The paper seems to have omitted a basic design in such studies for three of the experiments reported whilst the fourth one introduces an unnecessary complication.

If  participants in an experiment are given all conditions of an experiment (called a repeated measures design) one needs to avoid order of treatment effects. We can do this by counterbalancing the treatments.

For their experiment 1 Mitra and Crawley (M&C) could have divided their 23 pupils into 2 groups, randomly assign the first group to SOLE  (self-organised learning environment) treatment then individual treatment and randomly assign the second group to individual treatment then SOLE treatment. This would have avoided puzzlement of M&C about “the strange result of a test result improving over time with no formal inputs in the interim”.

Similarly the three tests in M&C’s experiment 2 could have been counterbalanced along with individual vs SOLE treatment. Hence there would have been less of a need for M&C to have invoked an “anomalous expansion of understanding” concept to explain why individual testing produced better performance than SOLE testing. The same criticism goes for experiment 3.

Experiment 4 seems the equivalent of a design variable car crash. Individual and group work are counterbalanced but the difficulty level of the reading text are not examined fully. That is a grade 3 text is used in one trial and a grade 5 in another trial. A better design would have included both grades in both trials. Group work has been shown to be more effective if the task is more complex so the best one can say is that M&C’s experiment 4 maybe shows that this is quite strong even with a weak experimental design!

M&C acknowledge their confusing results via this statement in the discussion section:

“Children seem to enjoy working on a ‘hard’ question, when there is no competition and when they are in groups, using the Internet. There is some indication that the opposite is also true, namely, that children do better individually at easy things than they do in groups.”

Finally the journal that Mitra and Crawley went with has a publisher with a less than stellar reputation.

Thanks for reading.

9 thoughts on “Some obvious notes on Mitra and Crawley (2014)

  1. Yes. The authors state that they wanted to ‘measure the effectiveness of learning in unsupervised environments created inside schools’. If they mean ‘to what extent is SOLE a more powerful technique than some other classroom based / teacher directed input?’ they would need to include that condition in the design and they don’t.

    Also, the two tests are inherently different (one is a group test, the other is an individual test) and so are not directly comparable in the way the authors present them for comparison. It would have been a better design to test individuals immediately after the SOLE session so that the scores could be directly comparable.

    Somewhat interesting publisher! However, Mitra has published elsewhere, and I have to say that if I were a primary school teacher (I’m not) I’d give it a go!

    1. hi Sue

      thanks for pointing out the issue with the tests

      i think if they used more of a qualitative approach (and abandoned quantitative measures) there would be more interesting analysis and discussions, you see a hint of that with some of the reported reactions from the kids

      the trouble a lot of people see with Mitra’s project is that it is packaged and promoted in a this-is-the-best-method-everything-from-the-past-is-to-be-rejected wrapping, a shame

      ta
      mura

      1. Hi,
        It’s difficult to find research reports that EAP students can be expected to read really critically but you have found one! It’s reasonably accessible (not too technical), of general interest (especially for students of education or in teacher training programmes) and there are plenty of points to follow up. I particularly like the way you probed the publication. There’s good material here. If anyone uses it it in an EAP course it would be good to hear how it went.

        Best of luck!
        Sue

      2. ah some nice criteria for an EAP paper discussion you set out Sue, maybe some readers will be tempted to use this as you suggest🙂

        ta
        mura

  2. Well done for taking the time to show up some of the problems with this sloppy report of a very badly-designed study. Vague research questions, no proper control groups, no proper report of the data, unjustified conclusions, etc., etc Zero reliability, zero validity.

  3. Sorry, I meant to add that I agree with Sue Argent that it looks like a great experiment in educational terms, and one that deserves proper studies and reports.

Penny for your thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s