A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics

Rachel Heyard, Samuel Pawel, Joris Frese, Bernhard Voelkl, Hanno Würbel, Sarah McCann, Leonhard Held, Kimberley E. PhD Wever, Helena Hartmann and Louise Townsin
Additional contact information
Samuel Pawel: University of Zurich
Hanno Würbel: University of Bern
Kimberley E. PhD Wever: Radboud university medical center
Helena Hartmann: University Hospital Essen

No apdxk, MetaArXiv from Center for Open Science

Abstract: *Background:* Reproducibility is recognized as essential to scientific progress and integrity. Replication studies and large-scale replication projects, aiming to quantify different aspects of reproducibility, have become more common. Since no standardized approach to measuring reproducibility exists,a diverse set of metrics has emerged and a comprehensive overview is needed. *Methods:* We conducted a scoping review to identify large-scale replication projects that used metrics and methodological papers that proposed or discussed metrics. The project list was compiled by the authors. For the methodological papers, we searched Scopus, MedLine, PsycINFO andEconLit. Records were screened in duplicate against predefined inclusion criteria. Demographic information on included records and information on reproducibility metrics used, suggested or discussed was extracted. *Results:* We identified 49 large-scale projects and 97 methodological papers, and extracted 50 metrics. The metrics were characterized based on type (formulas and/or statistical models, frameworks, graphical representations, studies and questionnaires, algorithms), input required, and appropriate application scenarios. Each metric addresses a distinct question. *Conclusions:* Our review provides a comprehensive resource in the form of a “live”, interactive table for future replication teams and meta-researchers, offering support in how to select the most appropriate metrics that are aligned with research questions and project goals.

Date: 2024-11-26
New Economics Papers: this item is included in nep-ppm and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6745e4637ab0b3495aac21b6/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:apdxk

DOI: 10.31219/osf.io/apdxk

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().