Rely (only) on the rigorous evidence” is bad advice

Lant Pritchett

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: A popular interpretation of “evidence‐based” decision‐making is “rely (only) on the rigorous evidence” (RORE) via “systematic” reviews that: use objective protocols to generating the potentially relevant papers from the literature; then filter those to retain only the small subset that provide impact estimates regarded as “rigorous”; and summarize only those estimates. I use two sets of cross‐country impact estimates—on wage gains for migrants and private school learning gains—to illustrate this seemingly attractive approach is both empirically and conceptually unsound. First, the cross‐country variation in the rigorous estimates of impact is very large, which implies the average(s) from a systematic review is of little predictive use. In both empirical examples the “systematic review of the rigorous estimates” approach leads to worse predictions of impact across countries than the naïve use of country‐specific ordinary least squares estimates. Second, I contrast a systematic review—RORE approach with an “understanding” approach—which seeks to encompass all of the available evidence into coherent understandings in forming judgments. In both examples the notion that the impact effects are constant across countries—“external validity”—is easily rejected. Insisting on privileged reliance on “rigorous” estimates in making context‐specific decisions is logically incoherent and deeply anti‐scientific.

Keywords: external validity; RCT (search for similar items in EconPapers)
JEL-codes: C10 C90 O10 (search for similar items in EconPapers)
Pages: 25 pages
Date: 2024-11-30
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published in Review of Development Economics, 30, November, 2024, 28(4), pp. 2034 - 2058. ISSN: 1363-6669

Downloads: (external link)
http://eprints.lse.ac.uk/119818/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:119818

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().