ArticleNature communications2025
Quantifying the reasoning abilities of LLMs on clinical cases.
Article in Nature communications, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 32 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
32 citing papers in PubMed.
- The Rise of Small Language Models in Healthcare: A Comprehensive Survey.Computer science review · 2026Article
- Entity-centric evaluation of large language model responses for medical question-answering tasks.PLOS digital health · 2026Article
- Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study.JMIR research protocols · 2026Article
- Large Language Model Performance on Multistep Clinical Cases: Comparative Study Across Question and Case Levels.JMIR medical education · 2026Article
- Large Language Model-Based Behavioral Activation Chatbot for Young People With Depression Using Artificial Users and Clinical Experts: Mixed Methods Evaluation.JMIR mental health · 2026Article
- Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases.Journal of medical Internet research · 2026Article
- Diagnostic Performance and Workup Efficiency of Large Language Models in Secondary Hypertension: A Blinded Comparative Study.Diagnostics (Basel, Switzerland) · 2026Article
- Large language models in respiratory care: safety and clinical integration.ERJ open research · 2026Article
- Gut Instincts, Machine Decisions: Evaluating AI Accuracy in the Diagnosis and Treatment of Disorders of Gut-Brain Interaction.Neurogastroenterology and motility · 2026Article
- Article
- Clinical Evaluation of the Clinical Reasoning Process of Large Language Models in Nephrology: Comparative Evaluation Study.JMIR formative research · 2026Article
- AI-PACE: a framework for integrating AI into medical education.NPJ digital medicine · 2026Review
- Advancing conversational diagnostic AI with multimodal reasoning.Nature medicine · 2026Article
- Article
- Open-Source Large Language Models and AI Health Equity: A Health Service Triangle Model Perspective.Journal of medical Internet research · 2026Article
- DxDirector: an agentic large language model driving the full-process clinical diagnosis.Nature communications · 2026Article
- The effects of multitype prompt engineering for large language models in hypertension treatment decisions.NPJ digital medicine · 2026Article
- Evaluating the clinical decision-making performance of large language models in clinically oriented thoracic anatomy scenarios: a comparative evaluation study.BMC medical education · 2026Article
- Leveraging Large Language Models for Automated Extraction of Abdominal Aortic Aneurysm Features from Radiology Reports.Diagnostics (Basel, Switzerland) · 2026Article
- Integrating Generative AI Into Patient-Centered Clinical Decision Support: Viewpoint on Research and Practice Considerations.Journal of medical Internet research · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Recent advances in reasoning-enhanced large language models (LLMs) show promise, yet their application in professional medicine, especially the evaluation of their reasoning process, remains underexplored. We present MedR-Bench, a benchmark of 1453 structured patient cases with reference reasoning derived from clinical case reports, spanning 13 body systems and 10 specialties across common and rare diseases. Our evaluation framework covers three stages of care: examination recommendation, diagnostic decision-making, and treatment planning. To assess reasoning quality, we develop the Reasoning Evaluator, an automated scorer of written reasoning along efficiency, factual accuracy, and completeness. We evaluate seven state-of-the-art reasoning LLMs. Here we show that current models exceed 85% accuracy on simple diagnostic tasks when sufficient examination results are available, but performance drops on examination recommendation and treatment planning. Reasoning is generally factual, yet critical steps are often missing. Open-source models are closing the gap with proprietary systems, highlighting potential for more accessible, equitable clinical AI.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.