Predicting the replicability of social science lab experiments
Adam Altmejd,
Anna Dreber,
Eskil Forsell,
Jürgen Huber,
Taisuke Imai,
Magnus Johannesson,
Michael Kirchler,
Gideon Nave and
Colin Camerer ()
Munich Reprints in Economics from University of Munich, Department of Economics
Abstract:
We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman rho of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to rho = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
Date: 2019
References: Add references at CitEc
Citations: View citations in EconPapers (9)
Downloads: (external link)
https://epub.ub.uni-muenchen.de/78212/1/78212.pdf (application/pdf)
Related works:
Journal Article: Predicting the replicability of social science lab experiments (2019) 
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
HTML/Text
Persistent link: https://EconPapers.repec.org/RePEc:lmu:muenar:78212
Access Statistics for this paper
More papers in Munich Reprints in Economics from University of Munich, Department of Economics Ludwigstr. 28, 80539 Munich, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Tamilla Benkelberg ().