Pre-Registered Analysis Plan: Is the Inter-Researcher Variability in Social Scientific Results Explicable? An Adversarial Collaboration and Joint Effort to Parse Model and Estimate Dispersion

Nate Breznau, Katrin Auspurg, Josef Brüderl, Felix Holzmeister, Gustav Nilsonne, Balazs Aczel, Cory J Clark, Hung Hoang Viet Nguyen, Marton Aron Varga and Eric Luis Uhlmann
Additional contact information
Nate Breznau: University of Bremen
Josef Brüderl: LMU Munich
Gustav Nilsonne: Karolinska Institutet
Balazs Aczel: Eotvos Lorand University

No h9t5c_v1, MetaArXiv from Center for Open Science

Abstract: We plan a collaborative research analysis of the original ‘many analysts’ study by Silberzahn et al.1 testing whether European soccer referees were skin-tone biased in giving red cards. Their study concluded that the answer has great model dependency, and in this sense is unclear. Auspurg and Brüderl2 re-analysed the methods and data and argued the answer is clearer when attending to causal inference and outliers. Researchers involved in the original study and follow-up agreed to an adversarial collaboration to better answer this pertinent research question and sources of variability in answers. They recruited other scholars to provide independent methodological and theoretical insights. The result is this study where together we jointly tackle the skin-tone bias hypothesis and the ‘parsing problem’ of identifying meaningful moderators that explain effect size dispersion across a multiverse of alternative specifications. We build on Auspurg and Brüderl’s proposal that analysts in the original study had very different interpretations of the conceptual research question, leading them to different empirical research questions, which in turn produced results that are not directly comparable. This pre-analysis plan describes our motivation, methods, and plan to contextualize our findings in larger discussions of race bias, multiverse analysis parsing, and subjectivity in science.

Date: 2023-05-29
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/646b0bece2673f200dee997f/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:h9t5c_v1

DOI: 10.31219/osf.io/h9t5c_v1

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().