ArticleNature communications2021
Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks.
Abstract read
In one paragraph
Article in Nature communications, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers.
0numbers the graph read from it
0cells of the map it votes in
15citing papers in PubMed
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
15 citing papers in PubMed.
- The rise of agentic AI in drug discovery from static tools to autonomous virtual scientists.npj drug discovery · 2026Review
- The Open Molecular Software Foundation (OMSF) and the Growing Role of Open Source Software in Molecular Modeling.Journal of chemical information and modeling · 2026Review
- The open-source Masala software suite: Facilitating rapid methods development for synthetic heteropolymer design.bioRxiv : the preprint server for biology · 2025Article
- Growing Glycans in Rosetta: Accurate de novo glycan modeling, density fitting, and rational sequon design.PLoS computational biology · 2024Article
- Modeling membrane geometries implicitly in Rosetta.Protein science : a publication of the Protein Society · 2024Article
- Combining machine learning with structure-based protein design to predict and engineer post-translational modifications of proteins.PLoS computational biology · 2024Article
- Implicit model to capture electrostatic features of membrane environment.PLoS computational biology · 2024Article
- NMR and Docking Calculations Reveal Novel Atomistic Selectivity of a Synthetic High-Affinity Free Fatty Acid vs. Free Fatty Acids in Sudlow's Drug Binding Sites in Human Serum Albumin.Molecules (Basel, Switzerland) · 2023Article
- How is Big Data reshaping preclinical aging research?Lab animal · 2023Review
- Simultaneous selection of nanobodies for accessible epitopes on immune cells in the tumor microenvironment.Nature communications · 2023Article
- Interpreting the molecular mechanisms of disease variants in human transmembrane proteins.Biophysical journal · 2023Article
- Towards generalizable predictions for G protein-coupled receptor variant expression.Biophysical journal · 2022Article
- The future of Microbial Biotechnology.Microbial biotechnology · 2022Article
- Diverse Scientific Benchmarks for Implicit Membrane Energy Functions.Journal of chemical theory and computation · 2021Article
- MHCEpitopeEnergy, a Flexible Rosetta-Based Biotherapeutic Deimmunization Platform.Journal of chemical information and modeling · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
50 authors.
Julia Koehler Leman *Center for Computational Biology, Flatiron Institute, Simons Foundation, New York, NY, 10010, USA. julia.koehler.leman@gmail.com.ORCID 0000-0002-5693-3593
Sergey Lyskov *Department of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA.ORCID 0000-0001-6380-6712
Steven M Lewis *Cyrus Biotechnology, 1201 Second Ave, Suite 900, Seattle, WA, 98101, USA.
Jared Adolf-BryfogleDepartment of Immunology and Microbiology, Scripps Research, La Jolla, CA, 92037, USA.ORCID 0000-0001-8586-1907
Rebecca F AlfordDepartment of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA.
Kyle BarlowGraduate Program in Bioinformatics, University of California San Francisco, San Francisco, CA, 94158, USA.ORCID 0000-0002-9787-0066
Ziv Ben-AharonDepartment of Microbiology and Molecular Genetics, Hebrew University, Hadassah Medical School, POB 12272, Jerusalem, 91120, Israel.ORCID 0000-0002-9623-6071
Daniel FarrellDepartment of Biochemistry, University of Washington, Seattle, WA, 98195, USA.ORCID 0000-0001-7024-7998
Jason FellGenome Center, University of California, Davis, CA, 95616, USA.ORCID 0000-0001-6680-2936
William A HansenDepartment of Chemistry and Chemical Biology, Rutgers, The State University of New Jersey, Piscataway, NJ, 08904, USA.
Ameya HarmalkarDepartment of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA.ORCID 0000-0001-6863-9634
Jeliazko JeliazkovProgram in Molecular Biophysics, Johns Hopkins University, Baltimore, MD, 21218, USA.ORCID 0000-0003-4249-1955
Georg KuenzeDepartment of Chemistry, Vanderbilt University, Nashville, TN, 37235, USA.
Justyna D KrysFaculty of Chemistry, Biological and Chemical Research Center, University of Warsaw, Pasteura 1, 02-093, Warsaw, Poland.ORCID 0000-0002-5318-8831
Ajasja LjubetičDepartment of Biochemistry, University of Washington, Seattle, WA, 98195, USA.ORCID 0000-0002-2171-1579
Amanda L LoshbaughDepartment of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA, 94158, USA.
Jack MaguireProgram in Bioinformatics and Computational Biology, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA.
Rocco MorettiDepartment of Chemistry, Vanderbilt University, Nashville, TN, 37235, USA.
Vikram Khipple MulliganCenter for Computational Biology, Flatiron Institute, Simons Foundation, New York, NY, 10010, USA.
Morgan L NanceProgram in Molecular Biophysics, Johns Hopkins University, Baltimore, MD, 21218, USA.ORCID 0000-0003-0997-9393
Phuong T NguyenDepartment of Physiology and Membrane Biology, School of Medicine, University of California, Davis, CA, 95616, USA.
Shane Ó ConchúirDepartment of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA, 94158, USA.
Shourya S Roy BurmanDepartment of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA.ORCID 0000-0001-9274-9104
Rituparna SamantaDepartment of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA.
Shannon T SmithCenter for Structural Biology, Vanderbilt University, Nashville, TN, 37235, USA.ORCID 0000-0002-1225-8928
Frank TeetsDepartment of Bioochemistry and Biophysics, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27516, USA.
Johanna K S TiemannLinderstrøm-Lang Centre for Protein Science, Department of Biology, University of Copenhagen, DK-2200, Copenhagen N., Denmark.ORCID 0000-0001-7551-6245
Andrew WatkinsDepartment of Biochemistry, Stanford University School of Medicine, Stanford, CA, 94305, USA.
Hope WoodsCenter for Structural Biology, Vanderbilt University, Nashville, TN, 37235, USA.ORCID 0000-0001-6615-545X
Brahm J YachninDepartment of Chemistry and Chemical Biology, Rutgers, The State University of New Jersey, Piscataway, NJ, 08904, USA.ORCID 0000-0001-6812-6329
Christopher D BahlInstitute for Protein Innovation, Boston, MA, 02115, USA.
Chris Bailey-KelloggDepartment of Computer Science, Dartmouth, Hanover, NH, 03755, USA.
David BakerDepartment of Biochemistry, University of Washington, Seattle, WA, 98195, USA.ORCID 0000-0001-7896-6217
Rhiju DasDepartment of Biochemistry, Stanford University School of Medicine, Stanford, CA, 94305, USA.ORCID 0000-0001-7497-0972
Frank DiMaioDepartment of Biochemistry, University of Washington, Seattle, WA, 98195, USA.
Sagar D KhareDepartment of Chemistry and Chemical Biology, Rutgers, The State University of New Jersey, Piscataway, NJ, 08904, USA.
Tanja KortemmeDepartment of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA, 94158, USA.
Jason W LabonteDepartment of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA.
Kresten Lindorff-LarsenLinderstrøm-Lang Centre for Protein Science, Department of Biology, University of Copenhagen, DK-2200, Copenhagen N., Denmark.ORCID 0000-0002-4750-6039
Jens MeilerDepartment of Chemistry, Vanderbilt University, Nashville, TN, 37235, USA.ORCID 0000-0001-8945-193X
William SchiefDepartment of Immunology and Microbiology, Scripps Research, La Jolla, CA, 92037, USA.ORCID 0000-0002-1120-0150
Ora Schueler-FurmanDepartment of Microbiology and Molecular Genetics, Hebrew University, Hadassah Medical School, POB 12272, Jerusalem, 91120, Israel.ORCID 0000-0002-1624-0362
Justin B SiegelGenome Center, University of California, Davis, CA, 95616, USA.
Amelie SteinLinderstrøm-Lang Centre for Protein Science, Department of Biology, University of Copenhagen, DK-2200, Copenhagen N., Denmark.ORCID 0000-0002-5862-1681
Vladimir Yarov-YarovoyDepartment of Physiology and Membrane Biology, School of Medicine, University of California, Davis, CA, 95616, USA.ORCID 0000-0002-2325-4834
Brian KuhlmanDepartment of Bioochemistry and Biophysics, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27516, USA.ORCID 0000-0003-4907-9699
Andrew Leaver-FayDepartment of Bioochemistry and Biophysics, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27516, USA.ORCID 0000-0003-4609-4340
Dominik GrontFaculty of Chemistry, Biological and Chemical Research Center, University of Warsaw, Pasteura 1, 02-093, Warsaw, Poland.
Jeffrey J GrayDepartment of Chemical and Biomolecular Engineering, Johns Hopkins University, Baltimore, MD, 21218, USA. jgray@jhu.edu.ORCID 0000-0001-6380-2324
Richard BonneauCenter for Computational Biology, Flatiron Institute, Simons Foundation, New York, NY, 10010, USA. bonneau@nyu.edu.ORCID 0000-0003-4354-7906
Funding
Workshop on Environmental Technology Transfer and EntrepreneurshipP42ES004699 · NIEHS · UNIVERSITY OF CALIFORNIA DAVIS · PI YANG, JUN · 1987 to 2021
$50.1MStructure based design of antibodies and vaccinesU19AI117905 · NIAID · VANDERBILT UNIVERSITY MEDICAL CENTER · PI CROWE, JAMES E · 2015 to 2019
$12.0MSTRUCTURAL BIOLOGY TRAINING PROGRAMT32GM008284 · NIGMS · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI KORTEMME, TANJA · 1988 to 2022
$10.1MPATHOGENESIS AND IMMUNOLOGY OF DISEASET32AI007244 · NIAID · SCRIPPS RESEARCH INSTITUTE, THE · PI NEMAZEE, DAVID · 1985 to 2022
$8.8MROSETTA An Integrated Protein Structure Modeling SuiteR01GM073151 · NIGMS · UNIV OF NORTH CAROLINA CHAPEL HILL · PI KUHLMAN, BRIAN A · 2005 to 2019
$7.8MNext-generation computational/chemical methods for complex RNA structuresR35GM122579 · NIGMS · STANFORD UNIVERSITY · PI Rhiju Das · 2017 to 2026
$7.2MStructure based design of trimer interface epitope focused universal influenza vaccinesU01AI150739 · NIAID · VANDERBILT UNIVERSITY MEDICAL CENTER · PI CROWE, JAMES E · 2020 to 2024
$6.1MGPU workstation for deep learning-based protein design and cryo-EM data processingR35GM131923 · NIGMS · UNIV OF NORTH CAROLINA CHAPEL HILL · PI BRIAN A KUHLMAN · 2019 to 2026
$6.1MIn silico Safety PharmacologyR01HL128537 · NHLBI · UNIVERSITY OF CALIFORNIA AT DAVIS · PI CLANCY, COLLEEN E, SANTANA, LUIS F · 2016 to 2024
$5.7MGlycan dependent epitopes of HIV broadly neutralizing antibodiesR01AI113867 · NIAID · SCRIPPS RESEARCH INSTITUTE, THE · PI PAULSON, JAMES C, SCHIEF, WILLIAM R. · 2014 to 2018
$4.8MPrediction of the Structure of Therapeutic Antibodies with their AntigensR01GM078221 · NIGMS · JOHNS HOPKINS UNIVERSITY · PI GRAY, JEFFREY J · 2006 to 2020
$4.0MFunctional Deimmunization of Therapeutic ProteinsR01GM098977 · NIGMS · DARTMOUTH COLLEGE · PI BAILEY-KELLOGG, CHRIS, GRISWOLD, KARL E · 2011 to 2019
$3.4MHoward Hughes Medical InstituteNCI NIH HHS F31 CA243353NCI NIH HHS F32 CA189246NCI NIH HHS R21 CA219847NHLBI NIH HHS R01 HL128537NIAID NIH HHS R01 AI113867NIAID NIH HHS R21 AI156570NIAID NIH HHS R56 AI110750NIAID NIH HHS T32 AI007244NIAID NIH HHS U01 AI150739NIAID NIH HHS U19 AI117905NIBIB NIH HHS R21 EB028342NIDA NIH HHS R01 DA046138NIEHS NIH HHS P42 ES004699NIGMS NIH HHS F31 GM123616NIGMS NIH HHS R01 GM073151NIGMS NIH HHS R01 GM076324NIGMS NIH HHS R01 GM078221NIGMS NIH HHS R01 GM080403NIGMS NIH HHS R01 GM098977NIGMS NIH HHS R01 GM110089NIGMS NIH HHS R01 GM121487NIGMS NIH HHS R01 GM123089NIGMS NIH HHS R01 GM127578NIGMS NIH HHS R01 GM129261NIGMS NIH HHS R01 GM132110NIGMS NIH HHS R01 GM132565NIGMS NIH HHS R35 GM122579NIGMS NIH HHS R35 GM131923NIGMS NIH HHS T32 GM008284NIGMS NIH HHS T32 GM067553NIH HHS S10 OD023680NINDS NIH HHS R01 NS103954NINDS NIH HHS UG3 NS114956
Abstract
Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.
Indexed as
Molecular Docking SimulationBenchmarkingBinding SitesHumansLigandsMacromolecular SubstancesProtein BindingProteinsReproducibility of ResultsSoftwareLigandsMacromolecular SubstancesProteins
Identifiers
What OpenQuestion holds
Textmetadata
LicenceCC BY
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.