Quantifying 'promising trials bias' in randomized controlled trials in education

Sam Sims (), Jake Anders, Matthew Inglis () and Hugues Lortie-Forgues ()
Additional contact information
Sam Sims: Centre for Education Policy and Equaliising Opportunities, UCL Institute of Education, University College London
Matthew Inglis: Centre for Mathematical Cognition, Loughborough University
Hugues Lortie-Forgues: Centre for Mathematical Cognition, Loughborough University

No 20-16, CEPEO Working Paper Series from UCL Centre for Education Policy and Equalising Opportunities

Abstract: Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results. We conduct a retrospective design analysis to quantify this bias across 23 promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the findings from the initial promising trial may simply have been exaggerated.

Keywords: randomized controlled trials; education; promising trials bias (search for similar items in EconPapers)
JEL-codes: C90 C93 I20 I21 (search for similar items in EconPapers)
Pages: 19 pages
Date: 2020-11, Revised 2020-11
New Economics Papers: this item is included in nep-edu and nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://repec-cepeo.ucl.ac.uk/cepeow/cepeowp20-16.pdf First version, 2020 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ucl:cepeow:20-16

Access Statistics for this paper

More papers in CEPEO Working Paper Series from UCL Centre for Education Policy and Equalising Opportunities Contact information at EDIRC.
Bibliographic data for series maintained by Jake Anders ().