Type I error rates are not usually inflated

Mark Rubin
Additional contact information
Mark Rubin: Durham University

No 3kv2b, MetaArXiv from Center for Open Science

Abstract: The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not inflate Type I error rates that are relevant to the statistical inferences that researchers usually make. I begin by introducing the concept of Type I error rates. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level and, in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be more a crisis of theoretical inference than of statistical inference.

Date: 2023-11-06
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6548c1efd45f5a0383e2fb6b/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:3kv2b

DOI: 10.31219/osf.io/3kv2b

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().