Replication does not measure scientific productivity

Joseph B Bak-Coleman, Richard P. Mann, Jevin West and Carl T. Bergstrom
Additional contact information
Joseph B Bak-Coleman: University of Washington

No rkyf7, SocArXiv from Center for Open Science

Abstract: Replication surveys are becoming a common tool for assessing the knowledge production of scientific disciplines. In psychology, economics, and preclinical cancer biology, replication rates near 50% have been argued as evidence the disciplines are not reliably producing knowledge, are rife with questionable research practices, and warrant reform. Concerns over failed replications have eroded faith in science, with claims that the vast majority of published research is false. However, these claims are often made under the assumption that effect sizes are fixed and point null hypotheses can be true in practice. Here we derive a theoretical model of the publication process that instead accounts for variation in observed effect sizes. We show that replication rates provide little insight into whether a scientific discipline is reliably and efficiently producing knowledge. In applying our model to data from a large-scale replication survey, we reveal that concerns over the reliability of scientific research may be overstated. Finally, we highlight how proposed reforms may be ineffective at improving replicability and can be detrimental to orthogonal measures of scientific productivity.

Date: 2022-04-28
New Economics Papers: this item is included in nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/62696b330890d80ba47b13b5/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:rkyf7

DOI: 10.31219/osf.io/rkyf7

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().