ArticleNPJ digital medicine2024
A framework for human evaluation of large language models in healthcare derived from literature review.
Article in NPJ digital medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT06997107 (AI-CARE), which is not on this map. Cited by 183 papers, 7 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
AI-CARE: AI to Create Accessible & Reliable Patient Education Materials
Who cites it
183 citing papers in PubMed, 7 syntheses or guidelines pooled it.
- Application of Large Language Models in Chronic Disease Care: Mixed Methods Systematic Review and Thematic Synthesis.Journal of medical Internet research · 2026Pooled it
- Research on patient-facing chatbots based on large language models in the care of older people: a living systematic review.European geriatric medicine · 2026Pooled it
- Testing and evaluation of generative large language models in electronic health record applications: a systematic review.Journal of the American Medical Informatics Association : JAMIA · 2026Pooled it
- Large language models in healthcare: applications, evaluation frameworks, and governance pathways - a scoping review and multidimensional framework.Frontiers in digital health · 2026Pooled it
- Evaluating Large Language Models in Ophthalmology: Systematic Review.Journal of medical Internet research · 2025Pooled it
- Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines.Journal of the American Medical Informatics Association : JAMIA · 2025Pooled it
- Data extraction methods for systematic review (semi)automation: Update of a living systematic review.F1000Research · 2021Pooled it
- Rethinking Pediatric Asthma Education Through Large Language Model Generation and Simplification: Randomized Double-Blind Study.Journal of medical Internet research · 2026Trial
- Entity-centric evaluation of large language model responses for medical question-answering tasks.PLOS digital health · 2026Article
- Evaluating the role of ChatGPT in patient questions regarding drug allergies.The World Allergy Organization journal · 2026Article
- Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus-based bone stress injury.Journal of experimental orthopaedics · 2026Article
- Large language models' performance in answering common patient questions about colonoscopy: an expert-based evaluation.BMC gastroenterology · 2026Observational
- Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial.JMIR mental health · 2026Article
- A Clinician-Centered Evaluation Framework for Large Language Models in Patient Education: Integrating the Technology Acceptance Model and Medical Condition Regard Scale.JMIR formative research · 2026Article
- Validating LLM judges for automated oversight of patient communication.medRxiv : the preprint server for health sciences · 2026Article
- Evaluation of large language model responses to patient questions on oral anticoagulant therapy: a comparative expert assessment.Exploratory research in clinical and social pharmacy · 2026Article
- The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review.Journal of medical Internet research · 2026Article
- Multiagent Large Language Model Framework for Psychotherapy Fidelity Assessment in Motivational Interviewing and Cognitive Behavioral Therapy Training: Cross-Sectional, Simulation-Based Evaluation Study.JMIR medical education · 2026Article
- Artificial Intelligence in Clinical Nutrition: Current Uses, Challenges, and Opportunities.Nutrients · 2026Review
- Methods of Evaluating Large Language Model-Based Health Care Applications Used by Nonprofessionals: Protocol for a Scoping Review.JMIR research protocols · 2026Article
123 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
15 authors.
Funding
Abstract
With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we propose QUEST, a comprehensive and practical framework for human evaluation of LLMs covering three phases of workflow: Planning, Implementation and Adjudication, and Scoring and Review. QUEST is designed with five proposed evaluation principles: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.