When is p-hacking detectable?

Stefan Faridani

Papers from arXiv.org

Abstract: Some forms of p-hacking cannot be detected by examining the t-curve (or p-curve). Standard tests may also fail to find even detectable forms of selective reporting. We propose a novel test that is consistent against every detectable form of p-hacking and remains interpretable even when the t-scores are not exactly normal. The test statistic is the distance between the smoothed empirical t-curve and the set of all distributions that would be possible in the absence of any selective reporting. This novel projection test can only be evaded in large meta-samples by selective reporting that also evades all other valid tests of restrictions on the t-curve. A second benefit of the projection test is that under the null hypothesis of no p-hacking we can check whether the projection residual could have been produced by other distortions not related to selective reporting, e.g. rounding and de-rounding. Applying the test to the Brodeur et al. (2020) meta-data, we find that the t-curves for RCTs, IVs, and DIDs are more distorted than could arise by chance. We confirm that these distortions cannot be explained by (de)rounding of t-scores or by the limited degrees of freedom of the underlying studies.

Date: 2025-06, Revised 2025-10
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2506.20035 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2506.20035

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().