ArticleNEJM AI2024
Almanac - Retrieval-Augmented Language Models for Clinical Medicine.
Article in NEJM AI, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 191 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
191 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Large Language Models in Colorectal Cancer Care and Clinical Decision Support: Systematic Review.Journal of medical Internet research · 2026Pooled it
- The TRIPOD-LLM reporting guideline for studies using large language models: a Korean translation.Ewha medical journal · 2025Guideline
- Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines.Journal of the American Medical Informatics Association : JAMIA · 2025Pooled it
- Large language models and conditional rules in clinical decision support systems.Health information science and systems · 2026Article
- How much do language and context affect the accuracy of AI health advice?BMJ health & care informatics · 2026Article
- Advancing Patient-Centred AI With Adaptive Machine Learning.Cancer medicine · 2026Review
- An Evidence-Based Framework for Patient-Facing Artificial Intelligence Integration in the Emergency Department.Journal of the American College of Emergency Physicians open · 2026Article
- Automated Brief Hospital Course Summarization in Cardiac Surgery Using a Lightweight Large Language Model-Based Framework: Development and Evaluation Study on the Medical Information Mart for Intensive Care-IV.Journal of medical Internet research · 2026Article
- LitBench: A Benchmark for Retrieval-Grounded Multi-Paper Evidence Synthesis in Epilepsy.medRxiv : the preprint server for health sciences · 2026Article
- Article
- Toward continuous quality observability in medicine.npj health systems · 2026Review
- Development and evaluation of a large language model-based, retrieval-augmented generation application for query response in early oncology clinical trials.ESMO real world data and digital oncology · 2026Article
- Ethical Considerations on Artificial Intelligence, Health and Health Care. All That Glisters Is Not Gold.European journal of clinical investigation · 2026Review
- Artificial intelligence in hepatocellular carcinoma diagnosis and AI-assisted treatment support: Current landscape and future directions.Liver research (Beijing, China) · 2026Review
- Guideline-augmented large language models for contraindication screening in interventional spine care.Interventional pain medicine · 2026Article
- Large Language Models for Patient Education in Cardiovascular Imaging: Prospective Observational Comparative Study.JMIR formative research · 2026Observational
- An autonomous multimodal AI agent for evidence-grounded ophthalmic diagnosis.Cell reports. Medicine · 2026Article
- Intelligent Framework for Adverse Drug Event Identification Using Large Language Models and Retrieval-Augmented Generation: Development and Evaluation Study.Journal of medical Internet research · 2026Article
- OntoCodex: a multi-agent biomedical ontology enrichment framework.npj health systems · 2026Article
- A Self-Controlled Benchmark of Retrieval-Augmented Generation for Large Language Models on Clinical Guideline Questions.Diagnostics (Basel, Switzerland) · 2026Article
131 more citing papers are in PubMed but not listed here.
Corrections and comments
- Update of
Authors and funding
22 authors.
Funding
Abstract
backgroundLarge language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements.
methodsWe tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties.
resultsAlmanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety.
conclusionsOur results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.