Ensuring the quality and specificity of preregistrations

Marjan Bakker, Coosje L S Veldkamp, Marcel A L M van Assen, Elise A V Crompvoets, How Hwee Ong, Brian A Nosek, Courtney K Soderberg, David Mellor and Jelte M Wicherts

PLOS Biology, 2020, vol. 18, issue 12, 1-18

Abstract: Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called “OSF Preregistration,” http://osf.io/prereg/). The Prereg Challenge format was a “structured” workflow with detailed instructions and an independent review to confirm completeness; the “Standard” format was “unstructured” with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the “structured” format restricted the opportunistic use of researcher degrees of freedom better (Cliff’s Delta = 0.49) than the “unstructured” format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.Researchers face many, often seemingly arbitrary choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. A study of two formats of preregistration available on the OSF reveals that the opportunistic use of researcher degrees of freedom aimed at obtaining statistical significance is restricted by using more extensive preregistration guidelines; however, these guidelines should be further improved.

Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000937 (text/html)
https://journals.plos.org/plosbiology/article/file ... 00937&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pbio00:3000937

DOI: 10.1371/journal.pbio.3000937

Access Statistics for this article

More articles in PLOS Biology from Public Library of Science
Bibliographic data for series maintained by plosbiology ().