Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, Colin Camerer (), Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, R. Alison Adcock, Paolo Avesani, Blazej M. Baczkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadège Bault, Derek Beaton, Julia Beitner, Roland G. Benoit, Ruud M. W. J. Berkers, Jamil P. Bhanji, Bharat B. Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine L. Bottenhorn, Alexander Bowring, Senne Braem, Hayley R. Brooks, Emily G. Brudner, Cristian B. Calderon, Julia A. Camilleri, Jaime J. Castrellon, Luca Cecchetti, Edna C. Cieslik, Zachary J. Cole, Olivier Collignon, Robert W. Cox, William A. Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles P. Davis, Alberto De Luca, Mauricio R. Delgado, Lysia Demetriou, Jeffrey B. Dennison, Xin Di, Erin W. Dickie, Ekaterina Dobryakova, Claire L. Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon B. Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Shiguang Fu, Adriana Galván, Remi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle J. Goeman, Sergej A. E. Golowin, Carlos González-García, Krzysztof J. Gorgolewski, Cheryl L. Grady, Mikella A. Green, João F. Guassi Moreira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson B. Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Chuan-Peng Hu, Scott A. Huettel, Matthew E. Hughes, Vittorio Iacovella, Alexandru D. Iordan, Peder M. Isager, Ayse I. Isik, Andrew Jahn, Matthew R. Johnson, Tom Johnstone, Michael J. E. Joseph, Anthony C. Juliano, Joseph W. Kable, Michalis Kassinopoulos, Cemal Koba, Xiang-Zhen Kong, Timothy R. Koscik, Nuri Erkut Kucukboyaci, Brice A. Kuhl, Sebastian Kupek, Angela R. Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Y. C. Li, Phui Cheng Lim, Evan N. Lintz, Schuyler W. Liphardt, Annabel B. Losecaat Vermeer, Bradley C. Love, Michael L. Mack, Norberto Malpica, Theo Marins, Camille Maumet, Kelsey McDonald, Joseph T. McGuire, Helena Melero, Adriana S. Méndez Leal, Benjamin Meyer, Kristin N. Meyer, Glad Mihai, Georgios D. Mitsis, Jorge Moll, Dylan M. Nielson, Gustav Nilsonne, Michael P. Notter, Emanuele Olivetti, Adrian I. Onicas, Paolo Papale, Kaustubh R. Patil, Jonathan E. Peelle, Alexandre Pérez, Doris Pischedda, Jean-Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia A. Reuter-Lorenz, Richard C. Reynolds, Emiliano Ricciardi, Jenny R. Rieck, Anais M. Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory R. Samanez-Larkin, Emilio Sanz-Morales, Margaret L. Schlichting, Douglas H. Schultz, Qiang Shen, Margaret A. Sheridan, Jennifer A. Silvers, Kenny Skagerlund, Alec Smith, David V. Smith, Peter Sokol-Hessner, Simon R. Steinkamp, Sarah M. Tashjian, Bertrand Thirion, John N. Thorp, Gustav Tinghög, Loreen Tisdall, Steven H. Tompson, Claudio Toro-Serey, Juan Jesus Torre Tresols, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna E. van ‘t Veer, Tom Verguts, Jean M. Vettel, Sagana Vijayarajah, Khoi Vo, Matthew B. Wall, Wouter D. Weeda, Susanne Weis, David J. White, David Wisniewski, Alba Xifra-Porxas, Emily A. Yearling, Sangsuk Yoon, Rui Yuan, Kenneth S. L. Yuen, Lei Zhang, Xu Zhang, Joshua E. Zosky, Thomas E. Nichols (), Russell A. Poldrack () and Tom Schonberg ()
Additional contact information
Rotem Botvinik-Nezer: Tel Aviv University
Juergen Huber: University of Innsbruck
Michael Kirchler: University of Innsbruck
Roni Iwanir: Tel Aviv University
Jeanette A. Mumford: University of Wisconsin–Madison
R. Alison Adcock: Duke University
Paolo Avesani: Fondazione Bruno Kessler
Blazej M. Baczkowski: Max Planck Institute for Human Cognitive and Brain Sciences
Aahana Bajracharya: Washington University in St. Louis
Leah Bakst: Boston University
Marco Barilari: Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain
Nadège Bault: University of Plymouth
Derek Beaton: Baycrest Health Sciences Centre
Julia Beitner: University of Amsterdam
Roland G. Benoit: Max Planck Institute for Human Cognitive and Brain Sciences
Ruud M. W. J. Berkers: Max Planck Institute for Human Cognitive and Brain Sciences
Jamil P. Bhanji: Rutgers University–Newark
Bharat B. Biswal: New Jersey Institute of Technology
Sebastian Bobadilla-Suarez: University College London
Tiago Bortolini: D’Or Institute for Research and Education (IDOR)
Katherine L. Bottenhorn: Florida International University
Alexander Bowring: University of Oxford
Senne Braem: Ghent University
Hayley R. Brooks: University of Denver
Emily G. Brudner: Rutgers University–Newark
Cristian B. Calderon: Ghent University
Julia A. Camilleri: Brain and Behaviour (INM-7), Research Centre Jülich
Jaime J. Castrellon: Duke University
Luca Cecchetti: IMT School for Advanced Studies Lucca
Edna C. Cieslik: Brain and Behaviour (INM-7), Research Centre Jülich
Zachary J. Cole: University of Nebraska–Lincoln
Olivier Collignon: University of Trento
Robert W. Cox: National Institutes of Health
William A. Cunningham: University of Toronto
Stefan Czoschke: Goethe University
Kamalaker Dadi: Inria, CEA, Université Paris-Saclay
Charles P. Davis: University of Connecticut
Alberto De Luca: University Medical Center Utrecht
Mauricio R. Delgado: Rutgers University–Newark
Lysia Demetriou: Imperial College London
Jeffrey B. Dennison: Temple University
Xin Di: New Jersey Institute of Technology
Erin W. Dickie: Centre for Addiction and Mental Health
Ekaterina Dobryakova: Kessler Foundation
Claire L. Donnat: Stanford University
Juergen Dukart: Brain and Behaviour (INM-7), Research Centre Jülich
Niall W. Duncan: Taipei Medical University
Joke Durnez: Stanford University
Amr Eed: Instituto de Neurociencias, CSIC-UMH
Simon B. Eickhoff: Brain and Behaviour (INM-7), Research Centre Jülich
Andrew Erhart: University of Denver
Laura Fontanesi: University of Basel
G. Matthew Fricke: University of New Mexico
Shiguang Fu: Zhejiang University of Technology
Adriana Galván: University of California Los Angeles
Remi Gau: Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain
Sarah Genon: Brain and Behaviour (INM-7), Research Centre Jülich
Tristan Glatard: Concordia University
Enrico Glerean: Aalto University
Jelle J. Goeman: Leiden University Medical Center
Sergej A. E. Golowin: Taipei Medical University
Carlos González-García: Ghent University
Krzysztof J. Gorgolewski: Stanford University
Cheryl L. Grady: Baycrest Health Sciences Centre
Mikella A. Green: Duke University
João F. Guassi Moreira: University of California Los Angeles
Olivia Guest: University College London
Shabnam Hakimi: Duke University
J. Paul Hamilton: Linköping University
Roeland Hancock: University of Connecticut
Giacomo Handjaras: IMT School for Advanced Studies Lucca
Bronson B. Harry: Western Sydney University
Colin Hawco: Centre for Addiction and Mental Health
Peer Herholz: McGill University
Gabrielle Herman: Centre for Addiction and Mental Health
Stephan Heunis: Eindhoven University of Technology
Felix Hoffstaedter: Brain and Behaviour (INM-7), Research Centre Jülich
Jeremy Hogeveen: University of New Mexico
Susan Holmes: Stanford University
Chuan-Peng Hu: Leibniz-Institut für Resilienzforschung (LIR)
Scott A. Huettel: Duke University
Matthew E. Hughes: Swinburne University of Technology
Vittorio Iacovella: University of Trento
Alexandru D. Iordan: University of Michigan
Peder M. Isager: Eindhoven University of Technology
Ayse I. Isik: Max Planck Institute for Empirical Aesthetics
Andrew Jahn: fMRI Laboratory, University of Michigan
Matthew R. Johnson: University of Nebraska–Lincoln
Tom Johnstone: Swinburne University of Technology
Michael J. E. Joseph: Centre for Addiction and Mental Health
Anthony C. Juliano: Kessler Foundation
Joseph W. Kable: University of Pennsylvania
Michalis Kassinopoulos: McGill University
Cemal Koba: IMT School for Advanced Studies Lucca
Xiang-Zhen Kong: Max Planck Institute for Psycholinguistics
Timothy R. Koscik: University of Iowa Carver College of Medicine
Nuri Erkut Kucukboyaci: Kessler Foundation
Brice A. Kuhl: University of Oregon
Sebastian Kupek: University of Innsbruck
Angela R. Laird: Florida International University
Claus Lamm: University of Vienna
Robert Langner: Brain and Behaviour (INM-7), Research Centre Jülich
Nina Lauharatanahirun: Human Research and Engineering Directorate
Hongmi Lee: Johns Hopkins University
Sangil Lee: University of Pennsylvania
Alexander Leemans: University Medical Center Utrecht
Andrea Leo: IMT School for Advanced Studies Lucca
Elise Lesage: Ghent University
Flora Li: Fralin Biomedical Research Institute
Monica Y. C. Li: University of Connecticut
Phui Cheng Lim: University of Nebraska–Lincoln
Evan N. Lintz: University of Nebraska–Lincoln
Schuyler W. Liphardt: University of New Mexico
Annabel B. Losecaat Vermeer: University of Vienna
Bradley C. Love: University College London
Michael L. Mack: University of Toronto
Norberto Malpica: Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos
Theo Marins: D’Or Institute for Research and Education (IDOR)
Camille Maumet: Inria, Univ Rennes, CNRS, Inserm, IRISA UMR 6074, Empenn ERL U 1228
Kelsey McDonald: Duke University
Joseph T. McGuire: Boston University
Helena Melero: Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos
Adriana S. Méndez Leal: University of California Los Angeles
Benjamin Meyer: Leibniz-Institut für Resilienzforschung (LIR)
Kristin N. Meyer: University of North Carolina at Chapel Hill
Glad Mihai: Max Planck Institute for Human Cognitive and Brain Sciences
Georgios D. Mitsis: McGill University
Jorge Moll: D’Or Institute for Research and Education (IDOR)
Dylan M. Nielson: National Institutes of Health
Gustav Nilsonne: Karolinska Institutet
Michael P. Notter: University Hospital Center and University of Lausanne
Emanuele Olivetti: Fondazione Bruno Kessler
Adrian I. Onicas: IMT School for Advanced Studies Lucca
Paolo Papale: IMT School for Advanced Studies Lucca
Kaustubh R. Patil: Brain and Behaviour (INM-7), Research Centre Jülich
Jonathan E. Peelle: Washington University in St. Louis
Alexandre Pérez: McGill University
Doris Pischedda: Charité Universitätsmedizin, corporate member of Freie Universität Berlin, Humboldt Universität zu Berlin, and Berlin Institute of Health
Jean-Baptiste Poline: McGill University
Yanina Prystauka: University of Connecticut
Shruti Ray: New Jersey Institute of Technology
Patricia A. Reuter-Lorenz: University of Michigan
Richard C. Reynolds: National Institutes of Health
Emiliano Ricciardi: IMT School for Advanced Studies Lucca
Jenny R. Rieck: Baycrest Health Sciences Centre
Anais M. Rodriguez-Thompson: University of North Carolina at Chapel Hill
Anthony Romyn: University of Toronto
Taylor Salo: Florida International University
Gregory R. Samanez-Larkin: Duke University
Emilio Sanz-Morales: Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos
Margaret L. Schlichting: University of Toronto
Douglas H. Schultz: University of Nebraska–Lincoln
Qiang Shen: Zhejiang University of Technology
Margaret A. Sheridan: University of North Carolina at Chapel Hill
Jennifer A. Silvers: University of California Los Angeles
Kenny Skagerlund: Linköping University
Alec Smith: Virginia Tech
David V. Smith: Temple University
Peter Sokol-Hessner: University of Denver
Simon R. Steinkamp: Research Centre Jülich
Sarah M. Tashjian: University of California Los Angeles
Bertrand Thirion: Inria, CEA, Université Paris-Saclay
John N. Thorp: Columbia University
Gustav Tinghög: Linköping University
Loreen Tisdall: Stanford University
Steven H. Tompson: Human Research and Engineering Directorate
Claudio Toro-Serey: Boston University
Juan Jesus Torre Tresols: Inria, CEA, Université Paris-Saclay
Leonardo Tozzi: Stanford University School of Medicine
Vuong Truong: Taipei Medical University
Luca Turella: University of Trento
Anna E. van ‘t Veer: Leiden University
Tom Verguts: Ghent University
Jean M. Vettel: US Combat Capabilities Development Command Army Research Laboratory
Sagana Vijayarajah: University of Toronto
Khoi Vo: Duke University
Matthew B. Wall: Invicro
Wouter D. Weeda: Leiden University
Susanne Weis: Brain and Behaviour (INM-7), Research Centre Jülich
David J. White: Swinburne University
David Wisniewski: Ghent University
Alba Xifra-Porxas: McGill University
Emily A. Yearling: University of Connecticut
Sangsuk Yoon: University of Dayton
Rui Yuan: Stanford University School of Medicine
Kenneth S. L. Yuen: Leibniz-Institut für Resilienzforschung (LIR)
Lei Zhang: University of Vienna
Xu Zhang: University of Connecticut
Joshua E. Zosky: University of Nebraska–Lincoln
Thomas E. Nichols: University of Oxford
Russell A. Poldrack: Stanford University
Tom Schonberg: Tel Aviv University

Nature, 2020, vol. 582, issue 7810, 84-88

Abstract: Abstract Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Date: 2020
References: Add references at CitEc
Citations: View citations in EconPapers (35)

Downloads: (external link)
https://www.nature.com/articles/s41586-020-2314-9 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:nature:v:582:y:2020:i:7810:d:10.1038_s41586-020-2314-9

Ordering information: This journal article can be ordered from
https://www.nature.com/

DOI: 10.1038/s41586-020-2314-9

Access Statistics for this article

Nature is currently edited by Magdalena Skipper

More articles in Nature from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().