Observing Many Researchers using the Same Data and Hypothesis Reveals a Hidden Universe of Data Analysis

Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Muna Adem, Jule Adriaans, Amalia Alvarez-Benjumea, Henrik Kenneth Andersen, Daniel Auer, Flavio Azevedo and Oke Bahnsen
Additional contact information
Nate Breznau: University of Bremen
Eike Mark Rinke: University of Leeds
Alexander Wuttke: University of Mannheim
Amalia Alvarez-Benjumea: Max Planck Institute for Research on collective goods
Flavio Azevedo: Cologne University
Authors registered in the RePEc Author Service: Jonathan Rogers

No cd5j9, MetaArXiv from Center for Open Science

Abstract: Findings from 162 researchers in 73 teams testing the same hypothesis with the same data reveal a universe of unique analytical possibilities leading to a broad range of results and conclusions. Surprisingly, the outcome variance mostly cannot be explained by variations in researchers’ modeling decisions or prior beliefs. Each of the 1,261 test models submitted by the teams was ultimately a unique combination of data-analytical steps. Because the noise generated in this crowdsourced research mostly cannot be explained using myriad meta-analytic methods, we conclude that idiosyncratic researcher variability is a threat to the reliability of scientific findings. This highlights the complexity and ambiguity inherent in the scientific data analysis process that needs to be taken into account in future efforts to assess and improve the credibility of scientific work.

Date: 2021-03-24
New Economics Papers: this item is included in nep-hme
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://osf.io/download/605b74e97d552b00ed1b750b/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:cd5j9

DOI: 10.31219/osf.io/cd5j9

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().