Metascientific reproducibility patterns revealed by informatic measure of knowledge

Daniele Fanelli

No 5vnhj, MetaArXiv from Center for Open Science

Abstract: Concerns for the irreproducibility of published findings are expressed in most scientific disciplines. However, current estimates of reproducibility suffer from theoretical and methodological limitations, including the assumptions that replication studies either “succeed” or “fail”, that they should yield the same results as the original study, and that “success” is determined by passing a P-value threshold or other arbitrary statistical criterion. These limitations ultimately hamper metascientists’ ability to measure, explain, compare and predict the reproducibility of results within and across scientific fields. A theory and methodology named “K theory” equates knowledge with information compression and offers a new paradigm to understand metascientific phenomena, including reproducibility. This study measured the theory’s key quantity, K, in original and replication studies in 22 data sets from 6 recent projects that measured the reproducibility of behavioural experiments (total N=867), and it tested three stringent predictions of K theory. In accordance with predictions, results show that: 1) average reproducibility, quantified as proportional knowledge loss, is less than 100%, it is lower in fields with more complex subject matters, and it decreases in pro- portion to measured divergences between original and replication study; 2) the log-transformed K values of original and replication study are strongly linearly associated; 3) irreproducibility is inversely related to the K of the original study. These patterns are observed at multiple levels of analysis - across primary studies, aggregations of studies and aggregations of fields - and yet were overlooked by past, ordinary analyses. The new method quantifies reproducibility as a continuous phenomenon, avoiding current limitations. It can also be used to identify determinants of irreproducibility and to predict the outcome of future replications. K theory’s success in revealing general reproducibility patterns support this theory’s potential to explain and predict other phenomena.

Date: 2020-03-31
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/5e833c801ab2430149e02659/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:5vnhj

DOI: 10.31219/osf.io/5vnhj

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().