Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Abel Brodeur, Nikolai Cook and Anthony Heyes

American Economic Review, 2020, vol. 110, issue 11, 3634-60

Abstract: The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal "revise and resubmit" process mitigates the problem; (iii) things are improving through time.

JEL-codes: A14 C12 C52 (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (130)

Downloads: (external link)
https://www.aeaweb.org/doi/10.1257/aer.20190687 (application/pdf)
https://doi.org/10.3886/E120246V1 (text/html)
https://www.aeaweb.org/doi/10.1257/aer.20190687.appx (application/pdf)
https://www.aeaweb.org/doi/10.1257/aer.20190687.ds (application/zip)
Access to full text is restricted to AEA members and institutional subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aea:aecrev:v:110:y:2020:i:11:p:3634-60

Ordering information: This journal article can be ordered from
https://www.aeaweb.org/journals/subscriptions

DOI: 10.1257/aer.20190687

Access Statistics for this article

American Economic Review is currently edited by Esther Duflo

More articles in American Economic Review from American Economic Association Contact information at EDIRC.
Bibliographic data for series maintained by Michael P. Albert ().