Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Alexandru Marcoci (), David P. Wilkinson, Ans Vercammen, Bonnie C. Wintle, Anna Lou Abatayo, Ernest Baskin, Henk Berkman, Erin M. Buchanan, Sara Capitán, Tabaré Capitán, Ginny Chan, Kent Jason G. Cheng, Tom Coupé, Sarah Dryhurst, Jianhua Duan, John E. Edlund, Timothy M. Errington, Anna Fedor, Fiona Fidler, James G. Field, Nicholas Fox, Hannah Fraser, Alexandra L. J. Freeman, Anca Hanea, Felix Holzmeister, Sanghyun Hong, Raquel Huggins, Nick Huntington-Klein, Magnus Johannesson, Angela M. Jones, Hansika Kapoor, John Kerr, Melissa Kline Struhl, Marta Kołczyńska, Yang Liu, Zachary Loomas, Brianna Luis, Esteban Méndez, Olivia Miske, Fallon Mody, Carolin Nast, Brian A. Nosek, E. Simon Parsons, Thomas Pfeiffer, W. Reed (), Jon Roozenbeek, Alexa R. Schlyfestone, Claudia R. Schneider, Andrew Soh, Zhongchen Song, Anirudh Tagat (), Melba Tutor, Andrew H. Tyner, Karolina Urbanska and Sander Linden
Additional contact information
Alexandru Marcoci: University of Cambridge
David P. Wilkinson: University of Melbourne
Ans Vercammen: University of Melbourne
Bonnie C. Wintle: University of Melbourne
Anna Lou Abatayo: Wageningen University and Research
Ernest Baskin: Saint Joseph’s University
Erin M. Buchanan: Harrisburg University of Science and Technology
Sara Capitán: Swedish University of Agricultural Sciences
Tabaré Capitán: Swedish University of Agricultural Sciences
Ginny Chan: Rhizom Psychological Services LLC
Kent Jason G. Cheng: The Pennsylvania State University
Tom Coupé: University of Canterbury
Sarah Dryhurst: University of Cambridge
Jianhua Duan: Statistics New Zealand
John E. Edlund: Rochester Institute of Technology
Timothy M. Errington: Center for Open Science
Anna Fedor: Independent researcher
Fiona Fidler: University of Melbourne
James G. Field: West Virginia University
Nicholas Fox: Center for Open Science
Hannah Fraser: University of Melbourne
Alexandra L. J. Freeman: University of Cambridge
Anca Hanea: University of Melbourne
Sanghyun Hong: University of Canterbury
Raquel Huggins: Harrisburg University of Science and Technology
Angela M. Jones: Texas State University
Hansika Kapoor: Monk Prayogshala
John Kerr: University of Cambridge
Melissa Kline Struhl: Massachusetts Institute of Technology
Marta Kołczyńska: Polish Academy of Sciences
Yang Liu: University of California, Santa Cruz
Zachary Loomas: Center for Open Science
Brianna Luis: Center for Open Science
Esteban Méndez: Central Bank of Costa Rica
Olivia Miske: Center for Open Science
Fallon Mody: University of Melbourne
Carolin Nast: University of Stavanger, School of Business and Law
Brian A. Nosek: Center for Open Science
E. Simon Parsons: Center for Open Science
Thomas Pfeiffer: Massey University
Jon Roozenbeek: University of Cambridge
Alexa R. Schlyfestone: Harrisburg University of Science and Technology
Claudia R. Schneider: University of Cambridge
Andrew Soh: University of Hawaii at Manoa
Zhongchen Song: New Zealand Institute of Economic Research (NZIER)
Melba Tutor: Independent researcher
Andrew H. Tyner: Center for Open Science
Karolina Urbanska: Independent researcher
Sander Linden: University of Cambridge

Nature Human Behaviour, 2025, vol. 9, issue 2, 287-304

Abstract: Abstract Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P

Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41562-024-01961-1 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:nathum:v:9:y:2025:i:2:d:10.1038_s41562-024-01961-1

Ordering information: This journal article can be ordered from
https://www.nature.com/nathumbehav/

DOI: 10.1038/s41562-024-01961-1

Access Statistics for this article

Nature Human Behaviour is currently edited by Stavroula Kousta

More articles in Nature Human Behaviour from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().