Computational Reproducibility in Finance: Evidence from 1,000 Tests

Christophe Pérignon, Olivier Akmansoy, Christophe Hurlin, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Albert Menkveld, Michael Razen and Utz Weitzel
Additional contact information
Christophe Pérignon: HEC Paris
Olivier Akmansoy: CNRS
Juergen Huber: University of Innsbruck
Michael Kirchler: University of Innsbruck
Michael Razen: University of Innsbruck
Utz Weitzel: VU University Amsterdam

No 1467, HEC Research Papers Series from HEC Paris

Abstract: We analyze the computational reproducibility of more than 1,000 empirical answers to six research questions in finance provided by 168 international research teams. Running the original researchers’ code on the same raw data regenerates exactly the same results only 52% of the time. Reproducibility is higher for researchers with better coding skills and for those exerting more effort. It is lower for more technical research questions, more complex code, and for results lying in the tails of the results distribution. Neither researcher seniority, nor peer-review ratings appear to be related to the level of reproducibility. Moreover, researchers exhibit strong overconfidence when assessing the reproducibility of their own research. We provide guidelines for finance researchers and discuss several implementable reproducibility policies for academic journals.

Keywords: computational reproducibility; open science; credibility of research; multi-analyst study; data-availability policy; scientific publishing (search for similar items in EconPapers)
JEL-codes: C80 C87 (search for similar items in EconPapers)
Pages: 61 pages
Date: 2022-04-06
References: Add references at CitEc
Citations:

Downloads: (external link)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4064172 Full text (text/html)

Related works:
Journal Article: Computational Reproducibility in Finance: Evidence from 1,000 Tests (2024)
Working Paper: Computational Reproducibility in Finance: Evidence from 1,000 Tests (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ebg:heccah:1467

DOI: 10.2139/ssrn.4064172

Access Statistics for this paper

More papers in HEC Research Papers Series from HEC Paris HEC Paris, 78351 Jouy-en-Josas cedex, France. Contact information at EDIRC.
Bibliographic data for series maintained by Antoine Haldemann ().