“Rely (only) on the rigorous evidence” is bad advice

Lant Pritchett

Review of Development Economics, 2024, vol. 28, issue 4, 2034-2058

Abstract: A popular interpretation of “evidence‐based” decision‐making is “rely (only) on the rigorous evidence” (RORE) via “systematic” reviews that: use objective protocols to generating the potentially relevant papers from the literature; then filter those to retain only the small subset that provide impact estimates regarded as “rigorous”; and summarize only those estimates. I use two sets of cross‐country impact estimates—on wage gains for migrants and private school learning gains—to illustrate this seemingly attractive approach is both empirically and conceptually unsound. First, the cross‐country variation in the rigorous estimates of impact is very large, which implies the average(s) from a systematic review is of little predictive use. In both empirical examples the “systematic review of the rigorous estimates” approach leads to worse predictions of impact across countries than the naïve use of country‐specific ordinary least squares estimates. Second, I contrast a systematic review—RORE approach with an “understanding” approach—which seeks to encompass all of the available evidence into coherent understandings in forming judgments. In both examples the notion that the impact effects are constant across countries—“external validity”—is easily rejected. Insisting on privileged reliance on “rigorous” estimates in making context‐specific decisions is logically incoherent and deeply anti‐scientific.

Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1111/rode.13037

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:rdevec:v:28:y:2024:i:4:p:2034-2058

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=1363-6669

Access Statistics for this article

Review of Development Economics is currently edited by E. Kwan Choi

More articles in Review of Development Economics from Wiley Blackwell
Bibliographic data for series maintained by Wiley Content Delivery ().