ArticleJournal of medical Internet research2024
Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4.
Article in Journal of medical Internet research, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 34 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
34 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- An LLM-enhanced AI agent for rapid diagnosis of acute aortic dissection in multi-center settings.iScience · 2026Article
- Large language models' performance in answering common patient questions about colonoscopy: an expert-based evaluation.BMC gastroenterology · 2026Observational
- Personalized Intelligent Chatbot Based on AI-Generated Content Assists Memoir Writing for Older Adults With Cognitive Impairment: Mixed Methods Study.JMIR human factors · 2026Article
- Comparative Performance of AI Models and Clinicians in Evidence-Based Cardiovascular Disease Management for People Living With HIV: Comparative Study.Journal of medical Internet research · 2026Article
- Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study.JMIR medical education · 2026Article
- Article
- A Comparative Analysis of GPT-4o and ERNIE Bot in a Chinese Radiation Oncology Exam.Journal of cancer education : the official journal of the American Association for Cancer Education · 2026Article
- Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board-style questions.JPGN reports · 2026Article
- Engaging Artificial Intelligence (AI)-based chatbots in digital health: A systematic review.PLOS digital health · 2026Article
- Can AI write your code? A case study of chatgpt's statistical coding capabilities for quantitative research.Health economics review · 2026Article
- Performance of Large Language Models in Analyzing Common Hypertension Scenarios.Hypertension (Dallas, Tex. : 1979) · 2026Article
- Assessment of the efficacy of ChatGPT responses to bacterial species-specific questions in microbiology.Access microbiology · 2026Article
- Concordance Between the Multidisciplinary Team and ChatGPT-4o Decisions: A Blinded, Cross-Sectional Concordance Study in Systemic Autoimmune Rheumatic Diseases.Diagnostics (Basel, Switzerland) · 2025Article
- A typology of physician input approaches to using AI chatbots for clinical decision-making.NPJ digital medicine · 2025Article
- Evaluating the Performance of DeepSeek-R1 and DeepSeek-V3 Versus OpenAI Models in the Chinese National Medical Licensing Examination: Cross-Sectional Comparative Study.JMIR medical education · 2025Article
- Retrieval-augmented generation for interpreting clinical laboratory regulations using large language models.Journal of pathology informatics · 2025Article
- Evaluation of Information Provided by ChatGPT Versions on Traumatic Dental Injuries for Dental Students and Professionals.Dental traumatology : official publication of International Association for Dental Traumatology · 2025Article
- Evaluating the role of large language models in traditional Chinese medicine diagnosis and treatment recommendations.NPJ digital medicine · 2025Article
- ChatGPT Performance Deteriorated in Patients with Comorbidities When Providing Cardiological Therapeutic Consultations.Healthcare (Basel, Switzerland) · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundArtificial intelligence, particularly chatbot systems, is becoming an instrumental tool in health care, aiding clinical decision-making and patient engagement.
objectiveThis study aims to analyze the performance of ChatGPT-3.5 and ChatGPT-4 in addressing complex clinical and ethical dilemmas, and to illustrate their potential role in health care decision-making while comparing seniors' and residents' ratings, and specific question types.
methodsA total of 4 specialized physicians formulated 176 real-world clinical questions. A total of 8 senior physicians and residents assessed responses from GPT-3.5 and GPT-4 on a 1-5 scale across 5 categories: accuracy, relevance, clarity, utility, and comprehensiveness. Evaluations were conducted within internal medicine, emergency medicine, and ethics. Comparisons were made globally, between seniors and residents, and across classifications.
resultsBoth GPT models received high mean scores (4.4, SD 0.8 for GPT-4 and 4.1, SD 1.0 for GPT-3.5). GPT-4 outperformed GPT-3.5 across all rating dimensions, with seniors consistently rating responses higher than residents for both models. Specifically, seniors rated GPT-4 as more beneficial and complete (mean 4.6 vs 4.0 and 4.6 vs 4.1, respectively; P<.001), and GPT-3.5 similarly (mean 4.1 vs 3.7 and 3.9 vs 3.5, respectively; P<.001). Ethical queries received the highest ratings for both models, with mean scores reflecting consistency across accuracy and completeness criteria. Distinctions among question types were significant, particularly for the GPT-4 mean scores in completeness across emergency, internal, and ethical questions (4.2, SD 1.0; 4.3, SD 0.8; and 4.5, SD 0.7, respectively; P<.001), and for GPT-3.5's accuracy, beneficial, and completeness dimensions.
conclusionsChatGPT's potential to assist physicians with medical issues is promising, with prospects to enhance diagnostics, treatments, and ethics. While integration into clinical workflows may be valuable, it must complement, not replace, human expertise. Continued research is essential to ensure safe and effective implementation in clinical environments.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.