Why rankings of biomedical image analysis competitions should be interpreted with care

Lena Maier-Hein (), Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P. Bradley, Aaron Carass, Carolin Feldmann, Alejandro F. Frangi, Peter M. Full, Bram Ginneken, Allan Hanbury, Katrin Honauer, Michal Kozubek, Bennett A. Landman, Keno März, Oskar Maier, Klaus Maier-Hein, Bjoern H. Menze, Henning Müller, Peter F. Neher, Wiro Niessen, Nasir Rajpoot, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Christian Stock, Danail Stoyanov, Abdel Aziz Taha, Fons Sommen, Ching-Wei Wang, Marc-André Weber, Guoyan Zheng, Pierre Jannin and Annette Kopp-Schneider
Additional contact information
Lena Maier-Hein: German Cancer Research Center (DKFZ)
Matthias Eisenmann: German Cancer Research Center (DKFZ)
Annika Reinke: German Cancer Research Center (DKFZ)
Sinan Onogur: German Cancer Research Center (DKFZ)
Marko Stankovic: German Cancer Research Center (DKFZ)
Patrick Scholz: German Cancer Research Center (DKFZ)
Tal Arbel: McGill University
Hrvoje Bogunovic: Medical University Vienna
Andrew P. Bradley: Queensland University of Technology
Aaron Carass: Johns Hopkins University
Carolin Feldmann: German Cancer Research Center (DKFZ)
Alejandro F. Frangi: The University of Leeds
Peter M. Full: German Cancer Research Center (DKFZ)
Bram Ginneken: Radboud University Center
Allan Hanbury: Institute of Information Systems Engineering, TU Wien
Katrin Honauer: Heidelberg University
Michal Kozubek: Masaryk University
Bennett A. Landman: Vanderbilt University
Keno März: German Cancer Research Center (DKFZ)
Oskar Maier: Universität zu Lübeck
Klaus Maier-Hein: German Cancer Research Center (DKFZ)
Bjoern H. Menze: Technical University of Munich
Henning Müller: Information System Institute, HES-SO
Peter F. Neher: German Cancer Research Center (DKFZ)
Wiro Niessen: Nuclear Medicine and Medical Informatics, Erasmus MC
Nasir Rajpoot: University of Warwick
Gregory C. Sharp: Massachusetts General Hospital
Korsuk Sirinukunwattana: University of Oxford
Stefanie Speidel: National Center for Tumor Diseases Dresden
Christian Stock: German Cancer Research Center (DKFZ)
Danail Stoyanov: University College London
Abdel Aziz Taha: Research Studios Austria FG
Fons Sommen: Eindhoven University of Technology
Ching-Wei Wang: National Taiwan University of Science and Technology
Marc-André Weber: University Medical Center Rostock
Guoyan Zheng: University of Bern
Pierre Jannin: LTSI (Laboratoire Traitement du Signal et de l’Image) - UMR_S 1099
Annette Kopp-Schneider: German Cancer Research Center (DKFZ)

Nature Communications, 2018, vol. 9, issue 1, 1-13

Abstract: Abstract International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Date: 2018
References: Add references at CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://www.nature.com/articles/s41467-018-07619-7 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:9:y:2018:i:1:d:10.1038_s41467-018-07619-7

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-018-07619-7

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().