ArticleDigital health
A two-stage evaluation of large language models for guideline-based Q&A and draft rehabilitation planning in stroke care.
Article in Digital health. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Objective: To evaluate large language models (LLMs) for stroke care in guideline-based question answering (Q&A) and individualized draft rehabilitation-plan generation, with explicit assessment of response quality, readability, and repeated-generation score stability. Methods: A two-stage study was conducted. Stage 1: Four LLMs generated best answers to 100 stroke-related questions; the top two models by accuracy advanced to Stage 2. In Stage 2a, ChatGPT-o3 and DeepSeek answered 20 guideline-derived clinical questions, each repeated three times. In Stage 2b, the two models generated individualized rehabilitation plans for 60 de-identified stroke cases, with three independent generations per case. Three senior clinicians evaluated outputs across correctness, completeness, readability, helpfulness, and safety using 5-point Likert scales. Chinese readability was assessed using the LDU-TGP platform, and English readability was assessed using the Flesch-Kincaid Grade Level. Generalized estimating equations were used for Stage 2a and Stage 2b comparisons to account for repeated generations clustered within clinical questions and patient cases, respectively. Results: In Stage 1, DeepSeek achieved the highest accuracy (91%), followed by ChatGPT-o3 (90%), Gemini (85%), and ChatGPT-4o (83%). In Stage 2a, no between-model differences across the five clinician-rated domains remained statistically significant after correction for multiple comparisons. ChatGPT-o3 generated clinical Q&A responses with a higher recommended reading age and English Flesch-Kincaid Grade Level, whereas the difference in Chinese Reading Difficulty Score did not remain significant after correction. In Stage 2b, ChatGPT-o3 significantly outperformed DeepSeek across correctness, completeness, readability, helpfulness, and safety in the GEE analysis. Repeated-generation score stability did not differ significantly between models. Conclusions: Within this single-center, expert-rated benchmark, ChatGPT-o3 and DeepSeek performed comparably across most guideline-based Q&A dimensions, whereas ChatGPT-o3 achieved higher clinician-rated scores for draft rehabilitation-plan generation. These findings do not establish clinical effectiveness. Prospective patient-centered validation and clinician-supervised review are required before implementation in routine stroke rehabilitation.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.