When is p-hacking detectable?

Stefan Faridani

Papers from arXiv.org

Abstract: We show that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or their p-values. Even when p-hacking is detectable, standard tests may lack power. We propose a novel test that detects every form of selective reporting that is detectable from the distribution of reported t-statistics. Our test statistic is the distance between the smoothed empirical t-curve and the set of possible honest distributions. This projection test is sharp and can only be evaded by selective reporting that also evades all other valid tests of restrictions on the t-curve. We also show how to avoid spurious rejections caused by some benign distortions in the t-curve. Applying the test to the Brodeur et al. (2020) meta-dataset, we find that the t-curves for RCTs and IVs are more distorted than could arise by chance, (de)rounding, or the Student-t approximation.

Date: 2025-06, Revised 2026-05
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2506.20035 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2506.20035

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().