Policy evaluation, randomized controlled trials, and external validity: A systematic review

Jörg Peters, Jörg Langbein and Gareth Roberts

No 589, Ruhr Economic Papers from RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen

Abstract: When properly implemented, Randomized Controlled Trials (RCT) can achieve a high degree of internal validity. Yet, if an RCT is to inform policy interventions that extend beyond the experimental population, it is critical to establish external validity. In this paper, we first present a theoretical framework of external validity and identify the potential hazards that compromise generalizing results beyond the studied population, namely Hawthorne effects, general equilibrium effects, specific sample problems, and special care in the provision of the randomized treatment. Second, we reviewed all RCTs published in leading economic journals between 2009 and 2014 and scrutinized the way they deal with external validity. Based on a set of objective indicators, we find that many published RCTs do not discuss hazards to external validity and do not provide the information that is necessary to assess potential problems. Apparently, external validity is not an important matter of concern during the peer review process. To conclude, we call for a more systematic approach to report the results of RCTs, including external validity dimensions.

Keywords: systematic review; internal validity; external validity; randomized controlled trials (search for similar items in EconPapers)
JEL-codes: C83 C93 (search for similar items in EconPapers)
Date: 2015
New Economics Papers: this item is included in nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (4)

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/123694/1/841211906.pdf (application/pdf)

Related works:
Journal Article: Policy evaluation, randomized controlled trials, and external validity—A systematic review (2016)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:rwirep:589

DOI: 10.4419/86788684

Access Statistics for this paper

More papers in Ruhr Economic Papers from RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen Contact information at EDIRC.
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics ().