Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

Lucas Coffman and Muriel Niederle ()

Journal of Economic Perspectives, 2015, vol. 29, issue 3, 81-98

Abstract: The social sciences—including economics—have long called for transparency in research to counter threats to producing robust and replicable results. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. They have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons. Hypothesis registries are a database of all projects attempted; the goal of this promising mechanism is to alleviate the "file drawer problem," which is that statistically significant results are more likely to be published, while other results are consigned to the researcher's "file drawer." Finally, we evaluate the efficacy of replications. We argue that even with modest amounts of researcher bias—either replication attempts bent on proving or disproving the published work, or poor replication attempts—replications correct even the most inaccurate beliefs within three to five replications. We offer practical proposals for how to increase the incentives for researchers to carry out replications.

JEL-codes: C38 (search for similar items in EconPapers)
Date: 2015
Note: DOI: 10.1257/jep.29.3.81
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (89)

Downloads: (external link)
http://www.aeaweb.org/articles.php?doi=10.1257/jep.29.3.81 (application/pdf)
Access to full text is restricted to AEA members and institutional subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aea:jecper:v:29:y:2015:i:3:p:81-98

Ordering information: This journal article can be ordered from
https://www.aeaweb.org/journals/subscriptions

Access Statistics for this article

Journal of Economic Perspectives is currently edited by Enrico Moretti

More articles in Journal of Economic Perspectives from American Economic Association Contact information at EDIRC.
Bibliographic data for series maintained by Michael P. Albert ().