Selective Reporting of Placebo Tests in Top Economics Journals

Anna Dreber, Magnus Johannesson and Yifan Yang

No 31, I4R Discussion Paper Series from The Institute for Replication (I4R)

Abstract: Placebo tests, where a null result is used to support the validity of the research design, is common in economics. Such tests provide an incentive to underreport statistically significant tests, a form of reversed p-hacking. Based on a pre-registered analysis plan, we test for such underreporting in all papers meeting our inclusion criteria (n=377) published in 11 top economics journals between 2009-2021. If the null hypothesis is true in all tests, 2.5% of them should be statistically significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]). Our results provide strong evidence of selective underreporting of statistically significant placebo tests in top economics journals.

Date: 2023
New Economics Papers: this item is included in nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/271197/1/I4R-DP031.pdf (application/pdf)

Related works:
Journal Article: Selective reporting of placebo tests in top economics journals (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:i4rdps:31

Access Statistics for this paper

More papers in I4R Discussion Paper Series from The Institute for Replication (I4R)
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics ().