Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Julia Koehler Leman (), Sergey Lyskov, Steven M. Lewis, Jared Adolf-Bryfogle, Rebecca F. Alford, Kyle Barlow, Ziv Ben-Aharon, Daniel Farrell, Jason Fell, William A. Hansen, Ameya Harmalkar, Jeliazko Jeliazkov, Georg Kuenze, Justyna D. Krys, Ajasja Ljubetič, Amanda L. Loshbaugh, Jack Maguire, Rocco Moretti, Vikram Khipple Mulligan, Morgan L. Nance, Phuong T. Nguyen, Shane Ó Conchúir, Shourya S. Roy Burman, Rituparna Samanta, Shannon T. Smith, Frank Teets, Johanna K. S. Tiemann, Andrew Watkins, Hope Woods, Brahm J. Yachnin, Christopher D. Bahl, Chris Bailey-Kellogg, David Baker, Rhiju Das, Frank DiMaio, Sagar D. Khare, Tanja Kortemme, Jason W. Labonte, Kresten Lindorff-Larsen, Jens Meiler, William Schief, Ora Schueler-Furman, Justin B. Siegel, Amelie Stein, Vladimir Yarov-Yarovoy, Brian Kuhlman, Andrew Leaver-Fay, Dominik Gront, Jeffrey J. Gray () and Richard Bonneau ()
Additional contact information
Julia Koehler Leman: Flatiron Institute, Simons Foundation
Sergey Lyskov: Johns Hopkins University
Steven M. Lewis: Cyrus Biotechnology
Jared Adolf-Bryfogle: Scripps Research
Rebecca F. Alford: Johns Hopkins University
Kyle Barlow: University of California San Francisco
Ziv Ben-Aharon: Hebrew University, Hadassah Medical School
Daniel Farrell: University of Washington
Jason Fell: University of California
William A. Hansen: Rutgers, The State University of New Jersey
Ameya Harmalkar: Johns Hopkins University
Jeliazko Jeliazkov: Johns Hopkins University
Georg Kuenze: Vanderbilt University
Justyna D. Krys: University of Warsaw, Pasteura 1
Ajasja Ljubetič: University of Washington
Amanda L. Loshbaugh: University of California San Francisco
Jack Maguire: University of North Carolina at Chapel Hill
Rocco Moretti: Vanderbilt University
Vikram Khipple Mulligan: Flatiron Institute, Simons Foundation
Morgan L. Nance: Johns Hopkins University
Phuong T. Nguyen: University of California
Shane Ó Conchúir: University of California San Francisco
Shourya S. Roy Burman: Johns Hopkins University
Rituparna Samanta: Johns Hopkins University
Shannon T. Smith: Vanderbilt University
Frank Teets: University of North Carolina at Chapel Hill
Johanna K. S. Tiemann: University of Copenhagen
Andrew Watkins: Stanford University School of Medicine
Hope Woods: Vanderbilt University
Brahm J. Yachnin: Rutgers, The State University of New Jersey
Christopher D. Bahl: Institute for Protein Innovation
Chris Bailey-Kellogg: Dartmouth
David Baker: University of Washington
Rhiju Das: Stanford University School of Medicine
Frank DiMaio: University of Washington
Sagar D. Khare: Rutgers, The State University of New Jersey
Tanja Kortemme: University of California San Francisco
Jason W. Labonte: Johns Hopkins University
Kresten Lindorff-Larsen: University of Copenhagen
Jens Meiler: Vanderbilt University
William Schief: Scripps Research
Ora Schueler-Furman: Hebrew University, Hadassah Medical School
Justin B. Siegel: University of California
Amelie Stein: University of Copenhagen
Vladimir Yarov-Yarovoy: University of California
Brian Kuhlman: University of North Carolina at Chapel Hill
Andrew Leaver-Fay: University of North Carolina at Chapel Hill
Dominik Gront: University of Warsaw, Pasteura 1
Jeffrey J. Gray: Johns Hopkins University
Richard Bonneau: Flatiron Institute, Simons Foundation

Nature Communications, 2021, vol. 12, issue 1, 1-15

Abstract: Abstract Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.nature.com/articles/s41467-021-27222-7 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:12:y:2021:i:1:d:10.1038_s41467-021-27222-7

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-021-27222-7

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().