ArticleJournal of the American Medical Informatics Association : JAMIA2024
Prompt engineering on leveraging large language models in generating response to InBasket messages.
Article in Journal of the American Medical Informatics Association : JAMIA, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Computational Use of Patient-Provider Secure Messaging Data to Achieve Better Clinical Efficiency and Quality of Communication: A Systematic Review.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2024Pooled it
- From patient notes to prognostication: The revolutionary potential of event-based foundation models in orthopaedics.Knee surgery, sports traumatology, arthroscopy : official journal of the ESSKA · 2026Article
- Evaluating Injection Laryngoplasty Skills Using a Foundation Model: A Feasibility Study.The Laryngoscope · 2026Article
- A Conceptual Model for Ambient AI Adoption: Perspectives From Academia and Industry.JMIR medical informatics · 2026Article
- Evidence, use cases, and implementation safeguards of large language models in primary care.Communications medicine · 2026Review
- A scoping review of studies on secure messaging through patient portals: persistent challenges and potential solutions.npj health systems · 2026Article
- Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI.NPJ digital medicine · 2026Review
- The Infectious Diseases Orchestrator: Embracing AI Literacy in the Agentic Era.Open forum infectious diseases · 2026Article
- Meeting the challenges of electronic health record (EHR) optimization.NPJ digital medicine · 2025Review
- Two-stage prompting framework with predefined verification steps for evaluating diagnostic reasoning tasks on two datasets.NPJ digital medicine · 2025Article
- Article
- Can Large Language Models Address Problem Gambling? Expert Insights from Gambling Treatment Professionals.Journal of gambling studies · 2025Article
- Utilization of Generative AI-drafted Responses for Managing Patient-Provider Communication.NPJ digital medicine · 2025Article
- Performance Evaluation of 18 Generative AI Models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese Pharmacist Licensing Examination: Comparative Study.JMIR medical education · 2025Article
- Confidence-linked and uncertainty-based staged framework for phenotype validation using large language models.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Use of a Medical Communication Framework to Assess the Quality of Generative Artificial Intelligence Replies to Primary Care Patient Portal Messages: Content Analysis.JMIR formative research · 2025Article
- A systematic review of early evidence on generative AI for drafting responses to patient messages.npj health systems · 2025Article
- Dall-E in hand surgery: Exploring the utility of ChatGPT image generation.Surgery open science · 2025Article
- Opportunities and risks of artificial intelligence in patient portal messaging in primary care.NPJ digital medicine · 2025Article
- Cross sectional pilot study on clinical review generation using large language models.NPJ digital medicine · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectivesLarge Language Models (LLMs) have been proposed as a solution to address high volumes of Patient Medical Advice Requests (PMARs). This study addresses whether LLMs can generate high quality draft responses to PMARs that satisfies both patients and clinicians with prompt engineering. MATERIALS AND
methodsWe designed a novel human-involved iterative processes to train and validate prompts to LLM in creating appropriate responses to PMARs. GPT-4 was used to generate response to the messages. We updated the prompts, and evaluated both clinician and patient acceptance of LLM-generated draft responses at each iteration, and tested the optimized prompt on independent validation data sets. The optimized prompt was implemented in the electronic health record production environment and tested by 69 primary care clinicians.
resultsAfter 3 iterations of prompt engineering, physician acceptance of draft suitability increased from 62% to 84% (P <.001) in the validation dataset (N = 200), and 74% of drafts in the test dataset were rated as "helpful." Patients also noted significantly increased favorability of message tone (78%) and overall quality (80%) for the optimized prompt compared to the original prompt in the training dataset, patients were unable to differentiate human and LLM-generated draft PMAR responses for 76% of the messages, in contrast to the earlier preference for human-generated responses. Majority (72%) of clinicians believed it can reduce cognitive load in dealing with InBasket messages. DISCUSSION AND
conclusionInformed by clinician and patient feedback synergistically, tuning in LLM prompt alone can be effective in creating clinically relevant and useful draft responses to PMARs.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.