Context Matters for Size: Why External Validity Claims and Development Practice do not Mix

Lant Pritchett and Justin Sandefur

Journal of Globalization and Development, 2014, vol. 4, issue 2, 161-197

Abstract: In this paper we examine how policymakers and practitioners should interpret the impact evaluation literature when presented with conflicting experimental and non-experimental estimates of the same intervention across varying contexts. As is well known, non-experimental estimates of a treatment effect comprise a causal treatment effect and a bias term due to endogenous selection into treatment. When non-experimental estimates vary across contexts any claim for external validity of an experimental result must make the assumption that (a) treatment effects are constant across contexts, while (b) selection processes vary across contexts. This assumption is rarely stated or defended in systematic reviews of evidence. As an illustration of these issues, we examine two thoroughly researched literatures in the economics of education – class size effects and gains from private schooling – which provide experimental and non-experimental estimates of causal effects from the same context and across multiple contexts. We show that the range of “true” causal effects in these literatures implies non-experimental estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. We conclude with recommendations for research and policy, including the need to evaluate programs in context, and avoid simple analogies to clinical medicine in which “systematic reviews” attempt to identify best-practices by putting most (or all) weight on the most “rigorous” evidence with no allowance for context.

Keywords: causal inference; external validity; policy evaluation; treatment effects (search for similar items in EconPapers)
Date: 2014
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (15)

Downloads: (external link)
https://doi.org/10.1515/jgd-2014-0004 (text/html)
For access to full text, subscription to the journal or payment for the individual article is required.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bpj:globdv:v:4:y:2014:i:2:p:161-197:n:5

Ordering information: This journal article can be ordered from
https://www.degruyter.com/journal/key/jgd/html

DOI: 10.1515/jgd-2014-0004

Access Statistics for this article

Journal of Globalization and Development is currently edited by Joseph E. Stiglitz, Kevin Gallagher, Jeronim Capaldo, Arjun Jayadev, José Antonio Ocampo and Dani Rodrik

More articles in Journal of Globalization and Development from De Gruyter
Bibliographic data for series maintained by Peter Golla ().