Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Sebastian Kranz and Peter Pütz

American Economic Review, 2022, vol. 112, issue 9, 3124-36

Abstract: Brodeur, Cook, and Heyes (2020) study hypothesis tests from economic articles and find evidence for p-hacking and publication bias, in particular for instrumental variable and difference-in-difference studies. When adjusting for rounding errors (introducing a novel method), statistical evidence for p-hacking from randomization tests and caliper tests at the 5 percent significance threshold vanishes for difference-in-difference studies but remains for instrumental variable studies. Results at the 1 percent and 10 percent significance thresholds remain largely similar. In addition, Brodeur, Cook, and Heyes derive latent distributions of z-statistics absent publication bias using two different approaches. We establish for each approach a result that challenges its applicability.

JEL-codes: A14 C12 C52 (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations: View citations in EconPapers (12)

Downloads: (external link)
https://www.aeaweb.org/doi/10.1257/aer.20210121 (application/pdf)
https://doi.org/10.3886/E159221V1 (text/html)
https://www.aeaweb.org/doi/10.1257/aer.20210121.appx (application/pdf)
https://www.aeaweb.org/doi/10.1257/aer.20210121.ds (application/zip)
Access to full text is restricted to AEA members and institutional subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aea:aecrev:v:112:y:2022:i:9:p:3124-36

Ordering information: This journal article can be ordered from
https://www.aeaweb.org/journals/subscriptions

DOI: 10.1257/aer.20210121

Access Statistics for this article

American Economic Review is currently edited by Esther Duflo

More articles in American Economic Review from American Economic Association Contact information at EDIRC.
Bibliographic data for series maintained by Michael P. Albert ().