ArticleJAMIA open2025
Uncertainty estimation in diagnosis generation from large language models: next-word probability is not pre-test probability.
Article in JAMIA open, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 14 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
14 citing papers in PubMed.
- Navigating uncertainty matters: Evaluating large language models for drug-drug interaction identification.Journal of managed care & specialty pharmacy · 2026Article
- Efficient and Effective Internal Memory Retrieval for LLM-Based Healthcare Prediction.Findings of ACL. ACL · 2026Article
- Proposed Context-of-Use Evaluation Framework for Medication Management Tasks Completed by Generative Artificial Intelligence.medRxiv : the preprint server for health sciences · 2026Article
- Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering.Journal of medical systems · 2026Article
- Safety-Constrained Agentic AI for Autism Screening: A Multimodal, Clinician-Guided Architecture.Cureus · 2026Article
- Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI.NPJ digital medicine · 2026Review
- A context-augmented large language model for accurate precision oncology medicine recommendations.Cancer cell · 2026Article
- Uncertainty-aware large language models for explainable disease diagnosis.NPJ digital medicine · 2025Article
- Using tournaments to calculate AUROC for zero-shot classification with LLMs.Findings of ACL. EMNLP. Conference on Empirical Methods in Natural Language Processing · 2025Article
- Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification.Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing · 2025Article
- Medicine Beyond Machines: Viewpoint on the Art of Thinking in the Age of AI.JMIR formative research · 2025Article
- Quo Vadis, AI-Empowered Doctor?JMIR medical education · 2025Article
- Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models.Findings of ACL. NAACL · 2025Article
- Research progress and implications of the application of large language model in shared decision-making in China's healthcare field.Frontiers in public health · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
Abstract
Objective: To evaluate large language models (LLMs) for pre-test diagnostic probability estimation and compare their uncertainty estimation performance with a traditional machine learning classifier. Materials and Methods: We assessed 2 instruction-tuned LLMs, Mistral-7B-Instruct and Llama3-70B-chat-hf, on predicting binary outcomes for Sepsis, Arrhythmia, and Congestive Heart Failure (CHF) using electronic health record (EHR) data from 660 patients. Three uncertainty estimation methods-Verbalized Confidence, Token Logits, and LLM Embedding+XGB-were compared against an eXtreme Gradient Boosting (XGB) classifier trained on raw EHR data. Performance metrics included AUROC and Pearson correlation between predicted probabilities. Results: The XGB classifier outperformed the LLM-based methods across all tasks. LLM Embedding+XGB showed the closest performance to the XGB baseline, while Verbalized Confidence and Token Logits underperformed. Discussion: These findings, consistent across multiple models and demographic groups, highlight the limitations of current LLMs in providing reliable pre-test probability estimations and underscore the need for improved calibration and bias mitigation strategies. Future work should explore hybrid approaches that integrate LLMs with numerical reasoning modules and calibrated embeddings to enhance diagnostic accuracy and ensure fairer predictions across diverse populations. Conclusions: LLMs demonstrate potential but currently fall short in estimating diagnostic probabilities compared to traditional machine learning classifiers trained on structured EHR data. Further improvements are needed for reliable clinical use.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.