ArticleNature methods2021
Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge.
Open access · hybridAbstract readValidation Study
In one paragraph
Article in Nature methods, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 65 papers.
0numbers the graph read from it
0cells of the map it votes in
65citing papers in PubMed
17.8field-weighted citation impact, top 1% of its field
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
65 citing papers in PubMed, 118 citations in OpenAlex.
- IHMValidation: Assessment of Integrative Structure Models Deposited to the Protein Data Bank.Journal of molecular biology · 2026Article
- DAQplugin: Interactive Deep Learning-Based Validation of Cryo-EM Protein Models in ChimeraX.bioRxiv : the preprint server for biology · 2026Article
- Cryogenic electron microscopy and coagulation factors.Current opinion in hematology · 2026Review
- A large-scale cryo-EM RNA motif dataset and benchmark for machine learning-based structure modeling.Computational biology and chemistry · 2026Article
- Comparative assessment of crystallographic and cryo-EM models in the Protein Data Bank.FEBS open bio · 2026Article
- KNexPHENIX: A PHENIX-Based Workflow for Improving Cryo-EM and Crystallographic Structural Models.Journal of chemical information and modeling · 2026Article
- A Large-Scale Cryo-EM RNA Motif Dataset and Benchmark for Machine Learning-Based Structure Modeling.bioRxiv : the preprint server for biology · 2026Article
- Cryo-EM Analysis in CASP16.Proteins · 2026Article
- Chart builder: an interactive tool for user driven data visualization in the electron microscopy data bank.Frontiers in bioinformatics · 2026Article
- The Inaugural Flatiron Institute Cryo-EM Conformational Heterogeneity Challenge.bioRxiv : the preprint server for biology · 2025Article
- Article
- KNexPHENIX: A PHENIX-Based Workflow for Improving Cryo-EM and Crystallographic Structural Models.bioRxiv : the preprint server for biology · 2025Article
- PCANN Program for Structure-Based Prediction of Protein-Protein Binding Affinity: Comparison With Other Neural-Network Predictors.Proteins · 2025Article
- Q-score as a reliability measure for protein, nucleic acid and small-molecule atomic coordinate models derived from 3DEM maps.Acta crystallographica. Section D, Structural biology · 2025Article
- AI-based quality assessment methods for protein structure models from cryo-EM.Current research in structural biology · 2025Review
- A functionally validated TCR-pMHC database for TCR specificity model development.bioRxiv : the preprint server for biology · 2025Article
- InstaMap: instant-NGP for cryo-EM density maps.Acta crystallographica. Section D, Structural biology · 2025Article
- CryoTEN: efficiently enhancing cryo-EM density maps using transformers.Bioinformatics (Oxford, England) · 2025Article
- Advancing structure modeling from cryo-EM maps with deep learning.Biochemical Society transactions · 2025Review
- AI-based methods for biomolecular structure modeling for Cryo-EM.Current opinion in structural biology · 2025Review
5 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
Authors and funding
54 authors at 20 institutions in 4 countries.
Catherine L LawsonInstitute for Quantitative Biomedicine, Rutgers, The State University of New Jersey, Piscataway, NJ, USA. cathy.lawson@rutgers.edu.ORCID http://orcid.org/0000-0002-3261-7035
Andriy KryshtafovychGenome Center, University of California, Davis, CA, USA.ORCID http://orcid.org/0000-0001-5066-7178
Paul D AdamsMolecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.ORCID http://orcid.org/0000-0001-9333-8219
Pavel V AfonineMolecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.
Matthew L BakerDepartment of Biochemistry and Molecular Biology, The University of Texas Health Science Center at Houston, Houston, TX, USA.ORCID http://orcid.org/0000-0001-9039-8523
Benjamin A BaradDepartment of Integrated Computational Structural Biology, The Scripps Research Institute, La Jolla, CA, USA.ORCID http://orcid.org/0000-0002-1016-862X
Paul BondYork Structural Biology Laboratory, Department of Chemistry, University of York, York, UK.ORCID http://orcid.org/0000-0002-8465-4823
Tom BurnleyScientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot, UK.ORCID http://orcid.org/0000-0001-5307-348X
Renzhi CaoDepartment of Computer Science, Pacific Lutheran University, Tacoma, WA, USA.ORCID http://orcid.org/0000-0002-8345-343X
Jianlin ChengDepartment of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO, USA.
Grzegorz ChojnowskiEuropean Molecular Biology Laboratory, c/o DESY, Hamburg, Germany.ORCID http://orcid.org/0000-0002-3796-8352
Kevin CowtanYork Structural Biology Laboratory, Department of Chemistry, University of York, York, UK.ORCID http://orcid.org/0000-0002-0189-1437
Ken A DillLaufer Center, Stony Brook University, Stony Brook, NY, USA.ORCID http://orcid.org/0000-0002-2390-2002
Frank DiMaioDepartment of Biochemistry and Institute for Protein Design, University of Washington, Seattle, WA, USA.
Daniel P FarrellDepartment of Biochemistry and Institute for Protein Design, University of Washington, Seattle, WA, USA.ORCID http://orcid.org/0000-0001-7024-7998
James S FraserDepartment of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA, USA.ORCID http://orcid.org/0000-0002-5080-2859
Mark A HerzikDepartment of Chemistry and Biochemistry, University of California, San Diego, La Jolla, CA, USA.ORCID http://orcid.org/0000-0001-6653-6682
Soon Wen HohYork Structural Biology Laboratory, Department of Chemistry, University of York, York, UK.ORCID http://orcid.org/0000-0003-1039-8000
Jie HouDepartment of Computer Science, Saint Louis University, St. Louis, MO, USA.ORCID http://orcid.org/0000-0002-8584-5154
Li-Wei HungLos Alamos National Laboratory, Los Alamos, NM, USA.
Maxim IgaevTheoretical and Computational Biophysics, Max Planck Institute for Biophysical Chemistry, Göttingen, Germany.ORCID http://orcid.org/0000-0001-8781-1604
Agnel P JosephScientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot, UK.ORCID http://orcid.org/0000-0002-0997-8422
Daisuke KiharaDepartment of Biological Sciences, Purdue University, West Lafayette, IN, USA.ORCID http://orcid.org/0000-0003-4091-6614
Dilip KumarVerna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX, USA.ORCID http://orcid.org/0000-0002-2721-678X
Sumit MittalBiodesign Institute, Arizona State University, Tempe, AZ, USA.ORCID http://orcid.org/0000-0002-5360-8947
Bohdan MonastyrskyyGenome Center, University of California, Davis, CA, USA.ORCID http://orcid.org/0000-0002-4803-0378
Mateusz OlekYork Structural Biology Laboratory, Department of Chemistry, University of York, York, UK.ORCID http://orcid.org/0000-0002-6347-9587
Colin M PalmerScientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot, UK.ORCID http://orcid.org/0000-0002-4883-1546
Ardan PatwardhanThe European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, UK.ORCID http://orcid.org/0000-0001-7663-9028
Alberto PerezDepartment of Chemistry, University of Florida, Gainesville, FL, USA.ORCID http://orcid.org/0000-0002-5054-5338
Jonas PfabDivision of Computing & Software Systems, University of Washington, Bothell, WA, USA.ORCID http://orcid.org/0000-0002-8285-571X
Grigore D PintilieDepartment of Bioengineering, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-0848-5335
Jane S RichardsonDepartment of Biochemistry, Duke University, Durham, NC, USA.ORCID http://orcid.org/0000-0002-3311-2944
Peter B RosenthalStructural Biology of Cells and Viruses Laboratory, Francis Crick Institute, London, UK.ORCID http://orcid.org/0000-0002-0387-2862
Daipayan SarkarDepartment of Biological Sciences, Purdue University, West Lafayette, IN, USA.ORCID http://orcid.org/0000-0002-4167-2108
Luisa U SchäferInstitute of Biological Information Processing (IBI-7: Structural Biochemistry) and Jülich Centre for Structural Biology (JuStruct), Forschungszentrum Jülich, Jülich, Germany.ORCID http://orcid.org/0000-0002-8481-6298
Michael F SchmidDivision of CryoEM and Biomaging, SSRL, SLAC National Accelerator Laboratory, Stanford University, Menlo Park, CA, USA.ORCID http://orcid.org/0000-0003-1077-5750
Gunnar F SchröderInstitute of Biological Information Processing (IBI-7: Structural Biochemistry) and Jülich Centre for Structural Biology (JuStruct), Forschungszentrum Jülich, Jülich, Germany.ORCID http://orcid.org/0000-0003-1803-5431
Mrinal ShekharBiodesign Institute, Arizona State University, Tempe, AZ, USA.
Dong SiDivision of Computing & Software Systems, University of Washington, Bothell, WA, USA.ORCID http://orcid.org/0000-0001-7039-2589
Abishek SingharoyBiodesign Institute, Arizona State University, Tempe, AZ, USA.ORCID http://orcid.org/0000-0002-9000-2397
Genki TerashiTheoretical and Computational Biophysics, Max Planck Institute for Biophysical Chemistry, Göttingen, Germany.ORCID http://orcid.org/0000-0002-5339-909X
Thomas C TerwilligerNew Mexico Consortium, Los Alamos, NM, USA.ORCID http://orcid.org/0000-0001-6384-0320
Andrea VaianaTheoretical and Computational Biophysics, Max Planck Institute for Biophysical Chemistry, Göttingen, Germany.ORCID http://orcid.org/0000-0002-8865-0651
Liguo WangDepartment of Biological Structure, University of Washington, Seattle, WA, USA.
Zhe WangThe European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, UK.ORCID http://orcid.org/0000-0003-1119-1765
Stephanie A WankowiczDepartment of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA, USA.
Christopher J WilliamsDepartment of Biochemistry, Duke University, Durham, NC, USA.
Martyn WinnScientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot, UK.ORCID http://orcid.org/0000-0003-0496-6796
Tianqi WuDepartment of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO, USA.
Xiaodi YuSMPS, Janssen Research and Development, Spring House, PA, USA.ORCID http://orcid.org/0000-0001-7520-0646
Kaiming ZhangDepartment of Bioengineering, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0003-0414-4776
Helen M BermanDepartment of Chemistry and Chemical Biology, Rutgers, The State University of New Jersey, Piscataway, NJ, USA.ORCID http://orcid.org/0000-0002-3337-0660
Wah ChiuDepartment of Bioengineering, Stanford University, Stanford, CA, USA. wahc@stanford.edu.ORCID http://orcid.org/0000-0002-8910-3078
Science and Technology Facilities Council · GBUniversity of York · GBMax Planck Institute for Biophysical Chemistry · DEUniversity of Washington · USDuke University · USEuropean Bioinformatics Institute · GBForschungszentrum Jülich · DELawrence Berkeley National Laboratory · USPurdue University West Lafayette · USRutgers, The State University of New Jersey · USSLAC National Accelerator Laboratory · USStanford University · USUniversity of California, Davis · USUniversity of California, San Francisco · USUniversity of Missouri · USUniversity of Washington Bothell · USArizona State University · USBarkatullah University · INBaylor College of Medicine · USBroad Institute · US
Funding
PROJECT 5 - DUKE - STRUCTURE VALIDATION AND IMPROVEMENT FOR PROTEINS AND N. ACIDSP01GM063210 · NIGMS · UNIVERSITY OF CALIF-LAWRENC BERKELEY LAB · PI ADAMS, PAUL DAVID · 2001 to 2021
$33.5MUnified Data Resource for Large Complexes Determined by Cryo-Electron MicroscopyR01GM079429 · NIGMS · STANFORD UNIVERSITY · PI CHIU, WAH · 2007 to 2021
$11.6MStructural Studies on RotavirusesR37AI036040 · NIAID · BAYLOR COLLEGE OF MEDICINE · PI PRASAD, BIDADI VENKATARAM · 2003 to 2022
$10.9MSTRUCTURAL BIOLOGY TRAINING PROGRAMT32GM008284 · NIGMS · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI KORTEMME, TANJA · 1988 to 2022
$10.1MFemtosecond nano-crystallography of membrane proteinsR01GM095583 · NIGMS · ARIZONA STATE UNIVERSITY-TEMPE CAMPUS · PI FROMME, PETRA · 2010 to 2023
$4.3MIntegrated Prediction of Protein Struture at 1D, 2D and 3D LevelsR01GM093123 · NIGMS · UNIVERSITY OF MISSOURI-COLUMBIA · PI CHENG, JIANLIN · 2010 to 2023
$3.8MSupplement to Resolving ensemble averaged conformations by multi-temperature x-ray crystallography - Equipment 2020R01GM123159 · NIGMS · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI FRASER, JAMES SOLOMON · 2018 to 2021
$2.3MExtending MolProbity Diagnosis & Healing Methods to Empower Better CryoEM & Xray Models at 2.5-4A Resolution, plus Versioned, Redeposited "GEMS" for Important Individual StructuresR35GM131883 · NIGMS · DUKE UNIVERSITY · PI RICHARDSON, DAVID CLAUDE · 2019 to 2022
$1.7MBuilding protein structure models for intermediate resolution cryo-electron microscopy mapsR01GM133840 · NIGMS · PURDUE UNIVERSITY · PI KIHARA, DAISUKE · 2020 to 2023
$1.6MStructural Modeling of Multifarious Protein ComplexesR01GM123055 · NIGMS · PURDUE UNIVERSITY · PI KIHARA, DAISUKE · 2017 to 2020
$1.2MBiotechnology and Biological Sciences Research Council BB/L006383/1Biotechnology and Biological Sciences Research Council BB/P000517/1Biotechnology and Biological Sciences Research Council BB/P000975/1Biotechnology and Biological Sciences Research Council BB/S005099/1Medical Research Council MR/N009614/1NIAID NIH HHS R37 AI036040NIGMS NIH HHS P01 GM063210NIGMS NIH HHS R01 GM079429NIGMS NIH HHS R01 GM095583NIGMS NIH HHS R01 GM123055NIGMS NIH HHS R01 GM123159NIGMS NIH HHS R01 GM133840NIGMS NIH HHS R35 GM131883Wellcome Trust 208398
Abstract
This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.
Indexed as
Models, MolecularCryoelectron MicroscopyCrystallography, X-RayProtein ConformationProteinsProteins
Identifiers
What OpenQuestion holds
Textmetadata
LicenceTDM
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.