Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It

Maximilian Kasy

Journal of Economic Perspectives, 2021, vol. 35, issue 3, 175-92

Abstract: A key challenge for interpreting published empirical research is the fact that published findings might be selected by researchers or by journals. Selection might be based on criteria such as significance, consistency with theory, or the surprisingness of findings or their plausibility. Selection leads to biased estimates, reduced coverage of confidence intervals, and distorted posterior beliefs. I review methods for detecting and quantifying selection based on the distribution of p-values, systematic replication studies, and meta-studies. I then discuss the conflicting recommendations regarding selection resulting from alternative objectives, in particular, the validity of inference versus the relevance of findings for decision-makers. Based on this discussion, I consider various reform proposals, such as deemphasizing significance, pre-analysis plans, journals for null results and replication studies, and a functionally differentiated publication system. In conclusion, I argue that we need alternative foundations of statistics that go beyond the single-agent model of decision theory.

JEL-codes: A14 C80 L82 (search for similar items in EconPapers)
Date: 2021
References: Add references at CitEc
Citations: View citations in EconPapers (19)

Downloads: (external link)
https://www.aeaweb.org/doi/10.1257/jep.35.3.175 (application/pdf)
https://www.aeaweb.org/doi/10.1257/jep.35.3.175.data (application/zip)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aea:jecper:v:35:y:2021:i:3:p:175-92

Ordering information: This journal article can be ordered from
https://www.aeaweb.org/journals/subscriptions

DOI: 10.1257/jep.35.3.175

Access Statistics for this article

Journal of Economic Perspectives is currently edited by Enrico Moretti

More articles in Journal of Economic Perspectives from American Economic Association Contact information at EDIRC.
Bibliographic data for series maintained by Michael P. Albert ().