EconPapers    
Economics at your fingertips  
 

Quantifying 'promising trials bias' in randomized controlled trials in education

Sam Sims (), Jake Anders (), Matthew Inglis () and Hugues Lortie-Forgues ()
Additional contact information
Sam Sims: Centre for Education Policy and Equaliising Opportunities, UCL Institute of Education, University College London
Jake Anders: Centre for Education Policy and Equaliising Opportunities, UCL Institute of Education, University College London
Matthew Inglis: Centre for Mathematical Cognition, Loughborough University
Hugues Lortie-Forgues: Centre for Mathematical Cognition, Loughborough University

No 20-16, CEPEO Working Paper Series from Centre for Education Policy and Equalising Opportunities, UCL Institute of Education

Abstract: Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results. We conduct a retrospective design analysis to quantify this bias across 23 promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the findings from the initial promising trial may simply have been exaggerated.Length: 19 pages

Keywords: randomized controlled trials; education; promising trials bias (search for similar items in EconPapers)
JEL-codes: I20 I21 C90 C93 (search for similar items in EconPapers)
Date: 2020-11, Revised 2020-11
References: Add references at CitEc
Citations: Track citations by RSS feed

Downloads: (external link)
https://repec-cepeo.ucl.ac.uk/cepeow/cepeowp20-16.pdf First version, 2020 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ucl:cepeow:20-16

Access Statistics for this paper

More papers in CEPEO Working Paper Series from Centre for Education Policy and Equalising Opportunities, UCL Institute of Education Contact information at EDIRC.
Bibliographic data for series maintained by Jake Anders ().

 
Page updated 2020-11-14
Handle: RePEc:ucl:cepeow:20-16