ArticleScientific reports2023
A vignette-based evaluation of ChatGPT's ability to provide appropriate and equitable medical advice across care contexts.
Article in Scientific reports, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 62 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
62 citing papers in PubMed, 2 syntheses or guidelines pooled it, 85 citations in OpenAlex.
- Large Language Models for Chatbot Health Advice Studies: A Systematic Review.JAMA network open · 2025Pooled it
- Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: A systematic review.Health promotion perspectives · 2023Pooled it
- Evaluating ChatGPT-4o as an Educational Support Tool for the Emergency Management of Dental Trauma: Randomized Controlled Study Among Students.JMIR medical education · 2025Trial
- An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation.PLOS digital health · 2026Article
- An analysis of AI platforms: Can AI provide genetics education?Journal of genetic counseling · 2026Article
- Comparison of three large language models in postoperative rehabilitation question answering after anterior cruciate ligament reconstruction based on expert ratings.BMC musculoskeletal disorders · 2026Article
- Generative Artificial Intelligence as a First Responder in Adolescent Testicular Torsion: A Case Report.Cureus · 2026Article
- Disclaimers and Referral Patterns for Medical Advice Across Urgency Levels: Large Language Model Evaluation Study.Journal of medical Internet research · 2026Article
- Predicting the Intention to Use Generative Artificial Intelligence for Health Information: Comparative Survey Study.Journal of medical Internet research · 2026Article
- Large Language Models in Cardiovascular Prevention: A Narrative Review and Governance Framework.Diagnostics (Basel, Switzerland) · 2026Review
- Clinical warrant in medical LLM evaluation.Frontiers in digital health · 2026Review
- Why do I use generative artificial intelligence (GenAI) to seek health information? A perceptual perspective of GenAI users.Frontiers in public health · 2026Article
- Large language model-assisted support among caregivers of patients with epilepsy: associations with caregiver burden and psychosocial outcomes.Frontiers in neurology · 2026Article
- Medical QA dialogue datasets in RAG systems performance evaluation and ChatGPT optimization.Scientific reports · 2025Article
- Large Language model (LLM) in temporomandibular disorder education: a comparative study.BMC oral health · 2025Article
- What artificial intelligence (AI) can tell us about Nasoalveolar Molding (NAM)?BMC oral health · 2025Article
- Can Large Language Models Address Problem Gambling? Expert Insights from Gambling Treatment Professionals.Journal of gambling studies · 2025Article
- Evaluation and Regulation of Artificial Intelligence Medical Devices for Clinical Decision Support.Annual review of biomedical data science · 2025Review
- Exploring ChatGPT's potential in diagnosing oral and maxillofacial pathologies: a study of 123 challenging cases.BMC oral health · 2025Article
- Assessment of Recommendations Provided to Athletes Regarding Sleep Education by GPT-4o and Google Gemini: Comparative Evaluation Study.JMIR formative research · 2025Article
2 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors at 2 institutions in 1 country.
Funding
Abstract
ChatGPT is a large language model trained on text corpora and reinforced with human supervision. Because ChatGPT can provide human-like responses to complex questions, it could become an easily accessible source of medical advice for patients. However, its ability to answer medical questions appropriately and equitably remains unknown. We presented ChatGPT with 96 advice-seeking vignettes that varied across clinical contexts, medical histories, and social characteristics. We analyzed responses for clinical appropriateness by concordance with guidelines, recommendation type, and consideration of social factors. Ninety-three (97%) responses were appropriate and did not explicitly violate clinical guidelines. Recommendations in response to advice-seeking questions were completely absent (N = 34, 35%), general (N = 18, 18%), or specific (N = 44, 46%). 53 (55%) explicitly considered social factors like race or insurance status, which in some cases changed clinical recommendations. ChatGPT consistently provided background information in response to medical questions but did not reliably offer appropriate and personalized medical advice.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.