The Illusion of Comparability Among Standardised Effect Sizes: Why Education Evaluations Should Report Raw Effects

Jack Rossiter, David Evans, Susannah Hares and Catherine Henny
Additional contact information
Jack Rossiter: Center for Global Development
Susannah Hares: Independent
Catherine Henny: Independent consultant

No 748, Working Papers from Center for Global Development

Abstract: Policymakers, practitioners, and donors rely on standardised effect sizes to compare education programmes and allocate resources. Yet in early-grade reading interventions in low- and middle-income countries, these metrics reflect differences in measurement tools, sample composition, and scaling choices as much as genuine learning gains. Drawing on data from 197 studies, we find that a single additional word per minute of reading fluency corresponds to anywhere from 0.03 to 0.55 standard deviations across studies. For tests designed by researchers—the most common type of test in this literature and the hardest to compare—a single correct response on a reading assessment can result in a standardised gain anywhere from 0.08 to 0.80 standard deviations. Converting raw to standardised effects also reorders which programmes appear most effective. Fewer than one in four papers (23 percent) present both standardised and raw effects, leaving readers unable to judge specifically what children can do differently as a result of the programme and whether that change matters educationally. We propose that researchers report raw effects and reference distributions alongside standardised estimates, that work on reading benchmarks be extended across more languages and countries, and that shared measurement frameworks be developed faster. Together, these reporting changes can improve interpretation for policymakers and allow evidence syntheses to explicitly model differences in test design and sample composition across studies. Each of these changes would help the field show what children actually gained from a programme, not just how large the effect appears.

Pages: 63 pages
Date: 2026-06-03
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.cgdev.org/publication/illusion-compara ... l&utm_campaign=repec

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cgd:wpaper:748

Access Statistics for this paper

More papers in Working Papers from Center for Global Development Contact information at EDIRC.
Bibliographic data for series maintained by Publications Manager ().