ArticleJournal of the American Medical Informatics Association : JAMIA2024
Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support.
Article in Journal of the American Medical Informatics Association : JAMIA, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 17 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
17 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Electronic health records-related determinants of healthcare professionals' burnout and mitigation strategies: systematic review and meta-analysis.Frontiers in public health · 2026Pooled it
- Evaluation of AI language models in answering pregnancy-related questions assessed by obstetrics specialists.Scientific reports · 2026Article
- Integrating rule-based NLP and large language models for statin information extraction from clinical notes.International journal of medical informatics · 2026Article
- Mapping applications and evaluations of LLM-enabled AI chatbots for health purposes: a scoping review.Frontiers in public health · 2026Article
- Comparison of rule- and large language model-based phenotype extraction from clinical notes for neurofibromatosis type 1.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Article
- Prompt Engineering in Clinical Practice: Tutorial for Clinicians.Journal of medical Internet research · 2025Article
- Optimizing Order Sets With a Large Language Model-Powered Multiagent System.JAMA network open · 2025Article
- Leveraging Retrieval-Augmented Large Language Models for Dietary Recommendations With Traditional Chinese Medicine's Medicine Food Homology: Algorithm Development and Validation.JMIR medical informatics · 2025Article
- Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- The Elements of Style for Interruptive Electronic Health Record Alerts.Applied clinical informatics · 2025Article
- Qualitative metrics from the biomedical literature for evaluating large language models in clinical decision-making: a narrative review.BMC medical informatics and decision making · 2024Review
- Pearls and Pitfalls for LLMs 2.0.Radiology · 2024Article
- Using large language model to guide patients to create efficient and comprehensive clinical care message.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Evaluating the Prevalence of Burnout Among Health Care Professionals Related to Electronic Health Record Use: Systematic Review and Meta-Analysis.JMIR medical informatics · 2024Review
- What can you do with a large language model?Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls.Applied clinical informatics · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
Abstract
objectivesTo evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts. MATERIALS AND
methodsWe extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness.
resultsFive CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001).
conclusionEnd-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.