What Is the False Discovery Rate in Empirical Research?

Tom Engsted

Econ Journal Watch, 2024, vol. 21, issue 1, 92–112

Abstract: A scientific discovery in empirical research, e.g., establishing a causal relationship between two variables, is typically based on rejecting a statistical null hypothesis of no relationship. What is the probability that such a rejection is a mistake? This probability is not controlled by the significance level of the test which is typically set at 5 percent. Statistically, the ‘False Discovery Rate’ (FDR) is the fraction of null rejections that are false. FDR depends on the significance level, the power of the test, and the prior probability that the null is true. All else equal, the higher the prior probability, the higher is the FDR. Economists have different views on how to assess this prior probability. I argue that for both statistical and economic reasons, the prior probability of the null should in general be quite high and, thus, the FDR in empirical economics is high, i.e., substantially higher than 5 percent. This may be a contributing factor behind the replication crisis that also haunts economics. Finally, I discuss conventional and newly proposed stricter significance thresholds and, more generally, the problems that passively observed economic and social science data pose for the traditional statistical testing paradigm.

Keywords: Hypothesis testing; false discoveries; replication crisis; prior probabilities; Bayesian analysis; statistical versus economic significance (search for similar items in EconPapers)
JEL-codes: B41 C11 C12 C18 C52 (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://econjwatch.org/File+download/1299/EngstedMar2024.pdf?mimetype=pdf (application/pdf)
https://econjwatch.org/1346 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ejw:journl:v:21:y:2024:i:1:p:92-112

Access Statistics for this article

Econ Journal Watch is currently edited by Daniel Klein

More articles in Econ Journal Watch from Econ Journal Watch Contact information at EDIRC.
Bibliographic data for series maintained by Jason Briggeman ().