The Reproducibility of Economics Research: A Case Study

Sylvérie Herbert, Hautahi Kingi, Flavio Stanchi and Lars Vilhubern

Working papers from Banque de France

Abstract: Given the importance of reproducibility for the scientific ethos, more and more journals have pushed for transparency of research through data availability policies. If the introduction and implementation of such data policies improve the availability of researchers' code and data, what is the impact on reproducibility? We describe and present the results of a large reproduction exercise in which we assess the reproducibility of research articles published in the American Economic Journal: Applied Economics, which has implemented a data availability policy since 2005. Our replication success rate is relatively moderate, with 37.78% of replication attempts successful. 68 of 162 eligible replication attempts successfully replicated the article's analysis (41.98%) conditional on non-confidential data. A further 69 (42.59%) were at least partially successful. A total of 98 out of 303 (32.34%) relied on confidential or proprietary data, and were thus not reproducible by this project. We also conduct several bibliometric analyses of reproducible vs. non-reproducible articles and show that replicable papers do not provide citation bonuses for authors.

Keywords: Replication; Reproducibility; Transparency; Replicability; Journal Policies (search for similar items in EconPapers)
JEL-codes: B41 C80 C81 C87 C88 (search for similar items in EconPapers)
Pages: 43 pages
Date: 2021
New Economics Papers: this item is included in nep-cwa, nep-hpe and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (4)

Downloads: (external link)
https://publications.banque-france.fr/sites/defaul ... ocuments/wp853_0.pdf

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bfr:banfra:853

Access Statistics for this paper

More papers in Working papers from Banque de France Banque de France 31 Rue Croix des Petits Champs LABOLOG - 49-1404 75049 PARIS. Contact information at EDIRC.
Bibliographic data for series maintained by Michael brassart ().