Evaluating meta-analysis as a replication success measure

Jasmine Muradchanian, Rink Hoekstra, Henk Kiers and Don van Ravenzwaaij
Additional contact information
Don van Ravenzwaaij: University of Groningen

No ax825, MetaArXiv from Center for Open Science

Abstract: The importance of replication in the social and behavioural sciences has been emphasized for decades. Various frequentist and Bayesian approaches have been proposed to qualify a replication study as successful or unsuccessful. One of them is meta-analysis. The focus of the present study is on the way meta-analysis functions as a replication success metric. To investigate this, original and replication studies that are part of two large-scale replication projects were used. For each original study, the probability of replication success was calculated using meta-analysis under different assumptions of the underlying population effect when replication results were unknown. The accuracy of the predicted overall replication success was evaluated once replication results became available using adjusted Brier scores. Our results showed that meta-analysis performed poorly when used as a replication success metric. In many cases, quantifying replication success using meta-analysis resulted in the conclusion where the replication was deemed a success regardless of the results of the replication study.

Date: 2023-06-19
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6490662d67aff803b4ee0152/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:ax825

DOI: 10.31219/osf.io/ax825

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().