Incentive-Compatible Critical Values

Adam McCloskey and Pascal Michaillat

No 29702, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Statistically significant results are more rewarded than insignificant ones, so researchers have the incentive to pursue statistical significance. Such p-hacking reduces the informativeness of hypothesis tests by making significant results much more common than they are supposed to be in the absence of true significance. To address this problem, we construct critical values of test statistics such that, if these values are used to determine significance, and if researchers optimally respond to these new significance standards, then significant results occur with the desired frequency. Such incentive-compatible critical values allow for p-hacking so they are larger than classical critical values. Using evidence from the social and medical sciences, we find that the incentive-compatible critical value for any test and any significance level is the classical critical value for the same test with approximately one fifth of the significance level—a form of Bonferroni correction. For instance, for a z-test with a significance level of 5%, the incentive-compatible critical value is 2.31 instead of 1.65 if the test is one-sided and 2.57 instead of 1.96 if the test is two-sided.

JEL-codes: C12 C18 (search for similar items in EconPapers)
Date: 2022-01
Note: AP DEV LS PE
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Published as Adam McCloskey & Pascal Michaillat, 2024. "Critical Values Robust to P-hacking," Review of Economics and Statistics.

Downloads: (external link)
http://www.nber.org/papers/w29702.pdf (application/pdf)

Related works:
Working Paper: Incentive-Compatible Critical Values (2022)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:29702

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w29702

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().