Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Reply

Abel Brodeur, Nikolai Cook and Anthony Heyes

American Economic Review, 2022, vol. 112, issue 9, 3137-39

Abstract: In Brodeur, Cook, and Heyes (2020) we present evidence that instrumental variable (and to a lesser extent difference-in-difference) articles are more p-hacked than randomized controlled trial and regression discontinuity design articles. We also find no evidence that (i) articles published in the top five journals are different; (ii) the "revise and resubmit" process mitigates the problem; (iii) things are improving through time. Kranz and Pütz (2022) apply a novel adjustment to address rounding errors. They successfully replicate our results with the exception of our shakiest finding: after adjusting for rounding errors, bunching of test statistics for difference-in-difference articles is now smaller around the 5 percent level (and coincidentally larger at the 10 percent level).

JEL-codes: A14 C12 C52 (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (6)

Downloads: (external link)
https://www.aeaweb.org/doi/10.1257/aer.20220277 (application/pdf)
https://doi.org/10.3886/E165621V1 (text/html)
https://www.aeaweb.org/doi/10.1257/aer.20220277.appx (application/pdf)
https://www.aeaweb.org/doi/10.1257/aer.20220277.ds (application/zip)
Access to full text is restricted to AEA members and institutional subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aea:aecrev:v:112:y:2022:i:9:p:3137-39

Ordering information: This journal article can be ordered from
https://www.aeaweb.org/journals/subscriptions

DOI: 10.1257/aer.20220277

Access Statistics for this article

American Economic Review is currently edited by Esther Duflo

More articles in American Economic Review from American Economic Association Contact information at EDIRC.
Bibliographic data for series maintained by Michael P. Albert ().