Evidence map›Paper›PMID 41559415›Full record

ArticleNature medicine2026

Holistic evaluation of large language models for medical tasks with MedHELM.

Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez and 73 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Nature medicine, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 40 papers, 2 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
40citing papers in PubMed, 2 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

40 citing papers in PubMed, 2 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Article
  4. Accountability for large language models in health care.Bulletin of the World Health Organization · 2026
    Article
  5. Article
  6. Article
  7. Article
  8. Article
  9. Review
  10. Review
  11. Article
  12. Observational
  13. Review
  14. Article
  15. Article
  16. Article
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

83 authors.

Suhana Bedi *Stanford University School of Medicine, Stanford, CA, USA. suhana@stanford.edu.ORCID http://orcid.org/0000-0002-4502-4191
Hejie Cui *Stanford University School of Medicine, Stanford, CA, USA.
Miguel Fuentes *Stanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0009-0002-1930-9529
Alyssa Unell *Stanford University School of Medicine, Stanford, CA, USA.
Michael WornowStanford University School of Medicine, Stanford, CA, USA.
Juan M BandaStanford Health Care, Palo Alto, CA, USA.ORCID http://orcid.org/0000-0001-8499-824X
Nikesh KotechaStanford Health Care, Palo Alto, CA, USA.
Timothy KeyesStanford Health Care, Palo Alto, CA, USA.ORCID http://orcid.org/0000-0003-0423-9679
Yifan MaiCenter for Research on Foundation Models (CRFM) & Department of Computer Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0009-0004-7270-2607
Mert OezMicrosoft Corporation, Redmond, WA, USA.
Hao QiuMicrosoft Corporation, Redmond, WA, USA.
Shrey JainMicrosoft Corporation, Redmond, WA, USA.
Leonardo SchettiniMicrosoft Corporation, Redmond, WA, USA.ORCID http://orcid.org/0009-0005-8373-7140
Mehr KashyapStanford University School of Medicine, Stanford, CA, USA.
Jason Alan FriesStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-9316-5768
Akshay SwaminathanStanford University School of Medicine, Stanford, CA, USA.
Philip ChungStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-1194-7510
Fateme Nateghi HaredashtStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-8874-8835
Ivan LopezStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0003-0246-2180
Asad AaliStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0009-0008-2120-5722
Gabriel TseStanford University School of Medicine, Stanford, CA, USA.
Ashwin NayakStanford University School of Medicine, Stanford, CA, USA.
Shivam VedakStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0009-0009-5034-0699
Sneha S JainStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-0592-1453
Birju PatelStanford University School of Medicine, Stanford, CA, USA.
Oluseyi FayanjuStanford University School of Medicine, Stanford, CA, USA.
Shreya ShahStanford University School of Medicine, Stanford, CA, USA.
Ethan GohStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0009-0001-7491-4257
Dong-Han YaoStanford University School of Medicine, Stanford, CA, USA.
Brian SoetiknoStanford University School of Medicine, Stanford, CA, USA.
Eduardo ReisStanford University School of Medicine, Stanford, CA, USA.
Sergios GatidisStanford University School of Medicine, Stanford, CA, USA.
Vasu DiviStanford University School of Medicine, Stanford, CA, USA.
Robson CapassoStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0003-2645-1793
Rachna SaralkarStanford University School of Medicine, Stanford, CA, USA.
Chia-Chun ChiangStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-7802-7172
Jenelle JindalStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-3959-5748
Tho PhamStanford University School of Medicine, Stanford, CA, USA.
Faraz GhoddusiStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0009-0005-1487-3237
Steven LinStanford University School of Medicine, Stanford, CA, USA.
Albert S ChiouStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-8436-0008
Hyo Jung HongStanford University School of Medicine, Stanford, CA, USA.
Mohana RoyStanford University School of Medicine, Stanford, CA, USA.
Michael F GensheimerStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-4897-3843
Hinesh PatelStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-7779-2910
Kevin SchulmanStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-8926-5085
Dev DashStanford University School of Medicine, Stanford, CA, USA.
Danton CharStanford University School of Medicine, Stanford, CA, USA.
Lance DowningStanford University School of Medicine, Stanford, CA, USA.
Francois GrolleauStanford University School of Medicine, Stanford, CA, USA.
Kameron BlackStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-3696-4050
Bethel MiesoStanford University School of Medicine, Stanford, CA, USA.
Aydin ZahedivashStanford University School of Medicine, Stanford, CA, USA.
Wen-Wai YimMicrosoft Corporation, Redmond, WA, USA.
Harshita SharmaMicrosoft Corporation, Redmond, WA, USA.ORCID http://orcid.org/0000-0003-4683-2606
Tony LeeCenter for Research on Foundation Models (CRFM) & Department of Computer Science, Stanford University, Stanford, CA, USA.
Hannah KirschStanford Health Care, Palo Alto, CA, USA.
Jennifer LeeStanford Health Care, Palo Alto, CA, USA.
Nerissa AmbersStanford Health Care, Palo Alto, CA, USA.ORCID http://orcid.org/0009-0008-9878-1824
Carlene LugtuStanford Health Care, Palo Alto, CA, USA.
Aditya SharmaStanford Health Care, Palo Alto, CA, USA.ORCID http://orcid.org/0009-0002-5351-9271
Bilal MawjiStanford Health Care, Palo Alto, CA, USA.
Alex AlekseyevStanford Health Care, Palo Alto, CA, USA.
Vicky ZhouStanford Health Care, Palo Alto, CA, USA.
Vikas KakkarStanford Health Care, Palo Alto, CA, USA.
Jarrod HelzerStanford Health Care, Palo Alto, CA, USA.
Anurang RevriStanford Health Care, Palo Alto, CA, USA.
Yair BannettStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-5134-3300
Roxana DaneshjouStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-7988-9356
Jonathan ChenStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-4387-8740
Emily AlsentzerStanford University School of Medicine, Stanford, CA, USA.
Keith MorseStanford University School of Medicine, Stanford, CA, USA.
Nirmal RavieHealth Africa Clinics, Kano, Nigeria.ORCID http://orcid.org/0000-0002-5146-2569
Nima AghaeepourStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-6117-8764
Vanessa KennedyStanford University School of Medicine, Stanford, CA, USA.
Akshay ChaudhariStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-3667-6796
Thomas WangStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0009-0000-1686-4178
Sanmi KoyejoCenter for Research on Foundation Models (CRFM) & Department of Computer Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-4023-419X
Matthew P LungrenStanford University School of Medicine, Stanford, CA, USA.
Eric HorvitzMicrosoft Corporation, Redmond, WA, USA.ORCID http://orcid.org/0000-0002-8823-0614
Percy LiangCenter for Research on Foundation Models (CRFM) & Department of Computer Science, Stanford University, Stanford, CA, USA.
Michael A PfefferStanford Health Care, Palo Alto, CA, USA.ORCID http://orcid.org/0000-0002-8958-0843
Nigam H ShahStanford University School of Medicine, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-9385-7158

Funding

Stanford Center for Clinical and Translational Research and EducationUM1TR004921 · NCATS · STANFORD UNIVERSITY · PI MANISHA DESAI, DEAN W FELSHER · 2024 to 2026
$30.1M
Center for Undiagnosed Diseases at StanfordU01NS134358 · NINDS · STANFORD UNIVERSITY · PI Jonathan Adam Bernstein, HOLLY K TABOR · 2023 to 2026
$3.1M
Measuring and Predicting Appropriate Antibiotic Use to Combat Resistant BacteriaR01AI178121 · NIAID · STANFORD UNIVERSITY · PI JONATHAN H. CHEN · 2023 to 2026
$3.1M
Novel Quality Measures for Primary Care Management of Attention-Deficit/Hyperactivity DisorderK23MH128455 · NIMH · STANFORD UNIVERSITY · PI Yair Bannett · 2022 to 2026
$970k
NCATS NIH HHS UM1 TR004921NIAID NIH HHS R01 AI178121NIMH NIH HHS K23 MH128455NINDS NIH HHS U01 NS134358
6 · The paper itself

Abstract

While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. Here we introduce MedHELM, an extensible evaluation framework with three contributions. First, a clinician-validated taxonomy organizing medical AI applications into five categories that mirror real clinical tasks-clinical decision support (diagnostic decisions, treatment planning), clinical note generation (visit documentation, procedure reports), patient communication (education materials, care instructions), medical research (literature analysis, clinical data analysis) and administration (scheduling, workflow coordination). These encompass 22 subcategories and 121 specific tasks reflecting daily medical practice. Second, a comprehensive benchmark suite of 37 evaluations covering all subcategories. Third, systematic comparison of nine frontier LLMs-Claude 3.5 Sonnet, Claude 3.7 Sonnet, DeepSeek R1, Gemini 1.5 Pro, Gemini 2.0 Flash, GPT-4o, GPT-4o mini, Llama 3.3 and o3-mini-using an automated LLM-jury evaluation method. Our LLM-jury uses multiple AI evaluators to assess model outputs against expert-defined criteria. Advanced reasoning models (DeepSeek R1, o3-mini) demonstrated superior performance with win rates of 66%, although Claude 3.5 Sonnet achieved comparable results at 15% lower computational cost. These results not only highlight current model capabilities but also demonstrate how MedHELM could enable evidence-based selection of medical AI systems for healthcare applications.

Indexed as

Large Language ModelsDecision Support Systems, ClinicalHumans

Identifiers

PMID41559415
PMCPMC13267972

What OpenQuestion holds

Textmetadata
LicenceTDM
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.