Reproducibility in Management Science

Miloš Fišar (), Ben Greiner (), Christoph Huber (), Elena Katok () and Ali I. Ozkes ()
Additional contact information
Miloš Fišar: Masaryk University Experimental Economics Laboratory, Department of Public Economics, Masaryk University, 60200 Brno, Czech Republic
Ben Greiner: Wirtschaftsuniversität Wien, Institute for Markets and Strategy, 1020 Vienna, Austria; University of New South Wales, School of Economics, Sydney NSW 2052, Australia
Christoph Huber: Wirtschaftsuniversität Wien, Institute for Markets and Strategy, 1020 Vienna, Austria
Elena Katok: Department of Operations Management, Naveen Jindal School of Management, University of Texas at Dallas, Richardson, Texas 75080
Ali I. Ozkes: SKEMA Business School, GREDEG, Université Côte d’Azur, Campus Grand Paris, 92156 Suresnes, France

Management Science, 2024, vol. 70, issue 3, 1343-1356

Abstract: With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness.

Keywords: reproducibility; replication; crowd science (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2023.03556 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:70:y:2024:i:3:p:1343-1356

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().