ArticleJAMA network open2025
Optimizing Order Sets With a Large Language Model-Powered Multiagent System.
Article in JAMA network open, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 3 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
3 citing papers in PubMed.
- Enhancing Physician Resilience to Generative AI: Multilevel Framework for Shared Authority, Verification, and Skill Preservation.Journal of medical Internet research · 2026Article
- From Agents to Governance: Essential AI Skills for Clinicians in the Large Language Model Era.Journal of medical Internet research · 2026Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
Abstract
Importance: Optimizing order sets is vital to enhance clinical decision support and improve patient care. Manual review is resource intensive and cannot timely identify potential improvements in order sets. Objective: To develop and evaluate the utility of a large language model (LLM)-powered multiagent system in optimizing order sets. Design, Setting, and Participants: A multiagent system was developed and evaluated between January 1, 2024, and December 31, 2024, which comprised agents for content critique, dynamic search, knowledge retrieval, medication verification, and suggestion summarization. A filter was developed to align suggestion usefulness scores with expert preferences. Experiment 1 evaluated 735 generated suggestions from a multiagent system developed for optimizing order sets, which were assessed by 3 physicians for 9 order sets and by 1 physician for 62 order sets. Experiment 2 implemented an LLM-as-a-judge approach to align generated suggestions with expert ratings and developed a filter to further refine the system's performance. The study was performed at Vanderbilt University Medical Center. A total of 735 suggestions for 71 order sets at VUMC were evaluated by 3 physicians. Main Outcomes and Measures: The ratings of accuracy, usefulness, feasibility, and impact; interrater agreement; and alignment against historical ordering data. Results: In evaluation 1 of experiment 1, the median values for the number of suggestions scoring 4 or higher at the order set level were 5 (IQR, 5-6) for the metrics of accuracy, 2 (IQR, 1-4) for usefulness, 1 (IQR, 0-3) for feasibility, and 1 (IQR, 0-2) for impact. Of 96 suggestions, 44 (46%; 95% CI, 36%-56%) aligned with historical ordering patterns. In evaluation 2 of experiment 1, 639 suggestions were generated for 62 order sets; 52 order sets had at least 1 useful suggestion, with a median of 2 (IQR, 1-3) useful suggestions. Overall, 122 suggestions (19%; 95% CI, 16%-22%) were rated as useful. After expert alignment, Cohen κ improved from 0.06 to 0.41. Filtering using the aligned scores reduced total suggestions by 29% while retaining 92% of useful suggestions. Conclusions and Relevance: In this cohort study of an LLM-powered multiagent system for optimizing order sets, leveraging LLMs and multiagent systems provided a scalable approach. Alignment with a small set of expert ratings significantly enhanced the LLM evaluation. Future research could refine reasoning capabilities and integrate useful suggestions into electronic health records, while engaging end-users as artificial intelligence-supported reviewers.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.