ArticleProceedings of machine learning research2024
Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges.
Article in Proceedings of machine learning research, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
21 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Mitigating hallucinations in healthcare AI: a systematic review of evidence-based strategies.BMC health services research · 2026Pooled it
- ChatGPT Versus DeepSeek in Resectability Categorization of Chronic Rhinosinusitis: A Comparison Based on EPOS 2020.World journal of otorhinolaryngology - head and neck surgery · 2026Article
- Using large language models to enhance clinically-driven missing data recovery algorithms in electronic health records.JAMIA open · 2026Article
- From BERT to GPT-4: A systematic review of AI-Driven toxicity extraction and grading in radiation oncology.Technical innovations & patient support in radiation oncology · 2026Article
- Cardiology-Chat: A Multi-LLMs Powered System for Cardiac Diagnostic Reasoning and Clinical Support.IEEE journal of translational engineering in health and medicine · 2026Article
- Evaluation of Large Language Models as Emergency Department Revisit Predictors.Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing · 2026Article
- Combining Clinician Expertise with Prompt Engineering enhances Small Language Models Reliability for Cancer Entity Recognition in Electronic Health Records.medRxiv : the preprint server for health sciences · 2025Article
- The In-depth Comparative Analysis of Four Large Language AI Models for Risk Assessment and Information Retrieval from Multi-Modality Prostate Cancer Work-up Reports.The world journal of men's health · 2025Article
- Generative Artificial Intelligence Summaries to Facilitate Emergency Department Handoff.Applied clinical informatics · 2025Article
- Benchmark evaluation of DeepSeek large language models in clinical decision-making.Nature medicine · 2025Article
- GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking.Proceedings of the conference. Association for Computational Linguistics. Meeting · 2025Article
- Author's Reply: Large Language Models Could Revolutionize Health Care, but Technical Hurdles May Limit Their Applications.Journal of medical Internet research · 2025Article
- Large language models in critical care.Journal of intensive medicine · 2025Review
- Unstructured Electronic Health Records of Dysphagic Patients Analyzed by Large Language Models.IEEE journal of translational engineering in health and medicine · 2025Article
- The potential of Generative Pre-trained Transformer 4 (GPT-4) to analyse medical notes in three different languages: a retrospective model-evaluation study.The Lancet. Digital health · 2025Article
- Article
- Using Large Language Models to Abstract Complex Social Determinants of Health From Original and Deidentified Medical Notes: Development and Validation Study.Journal of medical Internet research · 2024Article
- The challenges of integrating diverse data sources: A case study in major depression.Health services & outcomes research methodology · 2024Article
- Towards Reducing Diagnostic Errors with Interpretable Risk Prediction.Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting · 2024Article
- Using Large Language Models to Annotate Complex Cases of Social Determinants of Health in Longitudinal Clinical Records.medRxiv : the preprint server for health sciences · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
Abstract
Unstructured data in Electronic Health Records (EHRs) often contains critical information-complementary to imaging-that could inform radiologists' diagnoses. But the large volume of notes often associated with patients together with time constraints renders manually identifying relevant evidence practically infeasible. In this work we propose and evaluate a zero-shot strategy for using LLMs as a mechanism to efficiently retrieve and summarize unstructured evidence in patient EHR relevant to a given query. Our method entails tasking an LLM to infer whether a patient has, or is at risk of, a particular condition on the basis of associated notes; if so, we ask the model to summarize the supporting evidence. Under expert evaluation, we find that this LLM-based approach provides outputs consistently preferred to a pre-LLM information retrieval baseline. Manual evaluation is expensive, so we also propose and validate a method using an LLM to evaluate (other) LLM outputs for this task, allowing us to scale up evaluation. Our findings indicate the promise of LLMs as interfaces to EHR, but also highlight the outstanding challenge posed by "hallucinations". In this setting, however, we show that model confidence in outputs strongly correlates with faithful summaries, offering a practical means to limit confabulations.
Identifiers
39224857PMC11368037What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.