Over the past 20 years, randomised trials have become an increasingly important part of the research landscape in education. I've been part of that, particularly in work funded by the UK's EEF.
But have randomised trials delivered everything we wanted them to? No.
That's why, in new work with Sam Sims, Matthew Inglis, Hugo Lortie-Forgues, Ben Weidmann & Ben Styles, we're advocating for changes in how RCTs are used in education.
By Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues, Ben Styles and Ben Weidmann; Abstract: Over the last twenty years, education researchers have increasingly conducted randomised experiments with the goal of
A combination of small effects, high uncertainty, and effect variation means that randomised experiments in education have fallen short in their aim to inform directly the decisions of educators and policymakers.
We argue that this is because the goals of randomised experiments have become confused. We seek to clarify these, drawing important distinctions between experiments aiming to inform decision-makers and those testing theoretical models.
Many experiments aiming to inform decision-makers would be better replaced by rigorous quasi-experimental work or, where they are feasible, multi-site trials (individually randomised but carried out within multiple schools).
Experiments in education remain valuable — but especially to test theoretical models, which can then inform educators’ mental models or intervention design. These theory-informing experiments should be designed quite differently from those designed to inform decision-makers
A lot of money is spent conducting RCTs in education research.
We want to get the most out of this spending — and ensure the real challenges we identify don't provide a pretext for those who oppose RCTs, per se, from talking down their important role in education research.
Read our working paper and let us know what you think: https://econpapers.repec.org/RePEc:ucl:cepeow:23-07
We're still actively working on this, so your thoughts and suggestions are much appreciated.
By Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues, Ben Styles and Ben Weidmann; Abstract: Over the last twenty years, education researchers have increasingly conducted randomised experiments with the goal of
Over the last twenty years, education researchers have increasingly conducted randomised experiments with the goal of informing the decisions of educators and policymakers. Such experiments have generally employed broad, consequential, standardised outcome measures in the hope that this would allow decisionmakers to compare effectiveness of different approaches. However, a combination of small effect sizes, wide confidence intervals, and treatment effect heterogeneity means that researchers have largely failed to achieve this goal.
Just finished reading this. Great paper. You address many of the issues regarding the blanket 'gold standard' claims of RCTs and suggest more nuance. Thanks for this.
Jake Anders
Prof Prachi Srivastava