ArticleJournal of medical systems2023
Evaluating the Performance of Different Large Language Models on Health Consultation and Patient Education in Urolithiasis.
Article in Journal of medical systems, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 38 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
38 citing papers in PubMed, 73 citations in OpenAlex.
- Performance of 5 Large Language Models in Perioperative Consultation for Pediatric Hypospadias: Cross-Sectional Comparative Study.Journal of medical Internet research · 2026Article
- Guideline-based clinical reasoning in periodontology education: a comparative study of residents and large language models.BMC medical education · 2026Article
- Comparative study on the efficacy of large language models (LLMs) in health management for patients with urological and andrological diseases: a multidimensional quality evaluation and applicability analysis.World journal of urology · 2026Article
- Perspectives of audiologists in China on artificial intelligence in clinical practice and professional identity: a qualitative study.BMC medical education · 2026Article
- AI-driven Medical Care: Evaluation of Large Language Models in Generating Personalized Stroke Education Materials.Healthcare informatics research · 2026Article
- Comparison of the performance of large language models in answering patient questions related to cataract.iScience · 2026Article
- Evaluating Large Language Models as Medical Consultation Tools for Double Eyelid Surgery: A Cross-Language Study in English and Chinese.Aesthetic plastic surgery · 2026Article
- Effectiveness of large language models in preoperative and discharge education: a systematic review based on an evaluation framework.NPJ digital medicine · 2026Article
- Evaluating the accuracy and communication quality of large language models in Ewing sarcoma: a comparative analysis of ChatGPT, Claude, Gemini, DeepSeek, and Grok.Frontiers in pediatrics · 2026Article
- A Comparative Analysis of GPT-3.5, GPT-4, GPT-4 Omni, Gemini Advanced, and Gemini 1.5 in Answering Total Knee Replacement-Related Questions.Orthopaedic journal of sports medicine · 2026Article
- ZhongdaChat-ED: a medical large language model for personalized erectile dysfunction health consultation and professional clinical decision-making using retrieval-augmented generation.Asian journal of andrology · 2026Article
- TCMEval-PA: a question-answering benchmark dataset for the prescription audit of Traditional Chinese Medicine.Scientific data · 2025Article
- Efficacy of Large Language Models in Providing Evidence-Based Patient Education for Celiac Disease: A Comparative Analysis.Nutrients · 2025Article
- Evaluation of large language models in patient education for hyperthyroidism: A comparative study of chatgpt, gemini, and deepseek.Endocrine · 2025Article
- Feasibility analysis of generative artificial intelligence tools as a means of medical science popularization on urological diseases.Translational andrology and urology · 2025Article
- Large language model chatbots for patient education in kidney stones: a scoping review.World journal of urology · 2025Article
- Performance of a novel medical artificial intelligence large language model on supporting decision-making for emergency patients with suspected sepsis.World journal of emergency medicine · 2025Article
- Areas of research focus and trends in the research on the application of AIGC in healthcare.Journal of health, population, and nutrition · 2025Article
- Accuracy, readability, and understandability of large language models for prostate cancer information to the public.Prostate cancer and prostatic diseases · 2025Article
- What is the role of large language models in the management of urolithiasis?: a review.Urolithiasis · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors at 3 institutions in 1 country.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectivesTo evaluate the effectiveness of four large language models (LLMs) (Claude, Bard, ChatGPT4, and New Bing) that have large user bases and significant social attention, in the context of medical consultation and patient education in urolithiasis. MATERIALS AND
methodsIn this study, we developed a questionnaire consisting of 21 questions and 2 clinical scenarios related to urolithiasis. Subsequently, clinical consultations were simulated for each of the four models to assess their responses to the questions. Urolithiasis experts then evaluated the model responses in terms of accuracy, comprehensiveness, ease of understanding, human care, and clinical case analysis ability based on a predesigned 5-point Likert scale. Visualization and statistical analyses were then employed to compare the four models and evaluate their performance.
resultsAll models yielded satisfying performance, except for Bard, who failed to provide a valid response to Question 13. Claude consistently scored the highest in all dimensions compared with the other three models. ChatGPT4 ranked second in accuracy, with a relatively stable output across multiple tests, but shortcomings were observed in empathy and human caring. Bard exhibited the lowest accuracy and overall performance. Claude and ChatGPT4 both had a high capacity to analyze clinical cases of urolithiasis. Overall, Claude emerged as the best performer in urolithiasis consultations and education.
conclusionClaude demonstrated superior performance compared with the other three in urolithiasis consultation and education. This study highlights the remarkable potential of LLMs in medical health consultations and patient education, although professional review, further evaluation, and modifications are still required.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.