Context Matters for Size: Why External Validity Claims and Development Practice Don't Mix-Working Paper 336

Lant Pritchett ()
Authors registered in the RePEc Author Service: Lant Pritchett and Justin Sandefur

No 336, Working Papers from Center for Global Development

Abstract: In this paper we examine how policymakers and practitioners should interpret the impact evaluation literature when presented with conflicting experimental and non-experimental estimates of the same intervention across varying contexts. We show three things. First, as is well known, non-experimental estimates of a treatment effect comprise a causal treatment effect and a bias term due to endogenous selection into treatment. When non-experimental estimates vary across contexts any claim for external validity of an experimental result must make the assumption that (a) treatment effects are constant across contexts, while (b) selection processes vary across contexts. This assumption is rarely stated or defended in systematic reviews of evidence. Second, as an illustration of these issues, we examine two thoroughly researched literatures in the economics of education—class size effects and gains from private schooling—which provide experimental and non-experimental estimates of causal effects from the same context and across multiple contexts. We show that the range of “true” causal effects in these literatures implies OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Third, we show that in important cases in economics, parameter heterogeneity is driven by economy- or institution-wide contextual factors, rather than personal characteristics, making it difficult to overcome external validity concerns through estimation of heterogeneous treatment effects within a single localized sample. We conclude with recommendations for research and policy, including the need to evaluate programs in context, and avoid simple analogies to clinical medicine in which “systematic reviews” attempt to identify best-practices by putting most (or all) weight on the most “rigorous” evidence with no allowance for context.

Keywords: external validity; treatment effects; policy evaluation; causal inference (search for similar items in EconPapers)
JEL-codes: D04 I2 O2 (search for similar items in EconPapers)
Pages: 55 pages
Date: 2013-08
New Economics Papers: this item is included in nep-dev and nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (32)

Downloads: (external link)
http://www.cgdev.org/sites/default/files/context-matters-for-size_0.pdf

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cgd:wpaper:336

Access Statistics for this paper

More papers in Working Papers from Center for Global Development Contact information at EDIRC.
Bibliographic data for series maintained by Publications Manager ().