Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance

Christophe Pérignon (), Olivier Akmansoy, Christophe Hurlin, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johanneson, Michael Kirchler, Albert Menkveld, Michael Razen and Utz Weitzel
Additional contact information
Christophe Pérignon: GREGH - Groupement de Recherche et d'Etudes en Gestion à HEC - HEC Paris - Ecole des Hautes Etudes Commerciales - CNRS - Centre National de la Recherche Scientifique
Juergen Huber: Universität Innsbruck [Innsbruck]
Magnus Johanneson: SSE - Stockholm School of Economics
Michael Kirchler: Universität Innsbruck [Innsbruck]
Michael Razen: Universität Innsbruck [Innsbruck]
Utz Weitzel: VU - Vrije Universiteit Amsterdam [Amsterdam]

Working Papers from HAL

Abstract: We analyze the computational reproducibility of more than 1,000 empirical answers to six research questions in finance provided by 168 international research teams. Running the original researchers' code on the same raw data regenerates exactly the same results only 52% of the time. Reproducibility is higher for researchers with better coding skills and for those exerting more effort. It is lower for more technical research questions, more complex code, and for results lying in the tails of the results distribution. Neither researcher seniority, nor peer-review ratings appear to be related to the level of reproducibility. Moreover, researchers exhibit strong overconfidence when assessing the reproducibility of their own research. We provide guidelines for finance researchers and discuss several implementable reproducibility policies for academic journals.

Keywords: computational reproducibility; open science; credibility of research; multi-analyst study; data-availability policy; scientific publishing (search for similar items in EconPapers)
Date: 2022-10-11
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-03810013

DOI: 10.2139/ssrn.4064172

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().