Multiplicity Eludes Peer Review: The Case of COVID-19 Research

Oliver Gutiérrez-Hernández and Luis Ventura García
Additional contact information
Oliver Gutiérrez-Hernández: Department of Geography, University of Málaga, 29010 Málaga, Spain
Luis Ventura García: Institute of Natural Resources and Agrobiology of Seville (IRNAS), Spanish National Research Council (CSIC), 41012 Seville, Spain

IJERPH, 2021, vol. 18, issue 17, 1-10

Abstract: Multiplicity arises when data analysis involves multiple simultaneous inferences, increasing the chance of spurious findings. It is a widespread problem frequently ignored by researchers. In this paper, we perform an exploratory analysis of the Web of Science database for COVID-19 observational studies. We examined 100 top-cited COVID-19 peer-reviewed articles based on p -values, including up to 7100 simultaneous tests, with 50% including >34 tests, and 20% > 100 tests. We found that the larger the number of tests performed, the larger the number of significant results (r = 0.87, p < 10 ?6 ). The number of p -values in the abstracts was not related to the number of p -values in the papers. However, the highly significant results ( p < 0.001) in the abstracts were strongly correlated (r = 0.61, p < 10 ?6 ) with the number of p < 0.001 significances in the papers. Furthermore, the abstracts included a higher proportion of significant results (0.91 vs. 0.50), and 80% reported only significant results. Only one reviewed paper addressed multiplicity-induced type I error inflation, pointing to potentially spurious results bypassing the peer-review process. We conclude the need to pay special attention to the increased chance of false discoveries in observational studies, including non-replicated striking discoveries with a potentially large social impact. We propose some easy-to-implement measures to assess and limit the effects of multiplicity.

Keywords: multiple hypotheses testing; multiple testing problem; false discovery rate (FDR); environmental research; epidemiology; health geography; SARS-CoV-2 (search for similar items in EconPapers)
JEL-codes: I I1 I3 Q Q5 (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1660-4601/18/17/9304/pdf (application/pdf)
https://www.mdpi.com/1660-4601/18/17/9304/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jijerp:v:18:y:2021:i:17:p:9304-:d:628408

Access Statistics for this article

IJERPH is currently edited by Ms. Jenna Liu

More articles in IJERPH from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().