ArticleJournal of medical Internet research2023
Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument.
Article in Journal of medical Internet research, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 160 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
160 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Artificial Intelligence (AI) - Powered Documentation Systems in Healthcare: A Systematic Review.Journal of medical systems · 2025Pooled it
- Large Language Models for Chatbot Health Advice Studies: A Systematic Review.JAMA network open · 2025Pooled it
- Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review.Journal of medical Internet research · 2024Pooled it
- Trust in Generative AI for Health Information Consumption and the Effect of Learned Dependency: Randomized Controlled Experimental Study.Journal of medical Internet research · 2026Trial
- Comparing the performance of a large language model and naive human interviewers in interviewing children about a witnessed mock-event.PloS one · 2025Trial
- Evaluating the quality of artificial intelligence responses to psoriasis-related clinical and patient questions: a comparative study of ChatGPT, Gemini, and Microsoft Copilot.Proceedings (Baylor University. Medical Center) · 2026Article
- Factors Shaping Trust and Satisfaction With AI Medical Chatbots: A Mixed Methods Vignette Survey of Caregivers Seeking Guidance on Pediatric Infectious Diseases.Journal of medical Internet research · 2026Article
- Performance of DeepSeek-R1 and ChatGPT-5 in the Generation of North American Spine Society Clinical Guidelines for Adult Vertebral Compression Fractures: Comparative Study.Journal of medical Internet research · 2026Article
- Article
- Driving Revisit Intentions Through Medical Information and Service Quality in General Hospitals: An Extended Technology Acceptance Model Approach.Healthcare (Basel, Switzerland) · 2026Article
- Symptom burden in multiple long-term conditions: An AI-supported, mixed-methods concept elicitation study.JRSM open · 2026Article
- Performance of AI chatbots in answering questions on tooth bleaching: a multilingual comparative study.BMC oral health · 2026Article
- An expert-led benchmark using common patient questions: evaluating large language models for adolescent idiopathic scoliosis education.European spine journal : official publication of the European Spine Society, the European Spinal Deformity Society, and the European Section of the Cervical Spine Research Society · 2026Article
- Article
- Use of Commercially Available Large Language Models to Generate Information Leaflets on Post-Intensive Care Syndrome: Clinical Utility Assessment.JMIR formative research · 2026Article
- Development and Evaluation of an Augmented Artificial Intelligence Model for Urogynecology Queries.International urogynecology journal · 2026Article
- Artificial Intelligence Chatbots as Sources of Implant Dentistry Information for the Public: Validity and Reliability Assessment.European journal of dentistry · 2026Article
- The Impact of Generative Artificial Intelligence Use on Perceived English Learning Achievement: The Roles of Use Behavior and Task-Technology Fit.Behavioral sciences (Basel, Switzerland) · 2026Article
- Mapping AI startup investment and innovation in healthcare using a five-tier AI systems complexity framework.NPJ digital medicine · 2026Article
- Are AI chatbots ready for endodontics? Evaluating their validity, consistency, and readability in patient-oriented responses.Odontology · 2026Article
100 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundChatGPT-4 is the latest release of a novel artificial intelligence (AI) chatbot able to answer freely formulated and complex questions. In the near future, ChatGPT could become the new standard for health care professionals and patients to access medical information. However, little is known about the quality of medical information provided by the AI.
objectiveWe aimed to assess the reliability of medical information provided by ChatGPT.
methodsMedical information provided by ChatGPT-4 on the 5 hepato-pancreatico-biliary (HPB) conditions with the highest global disease burden was measured with the Ensuring Quality Information for Patients (EQIP) tool. The EQIP tool is used to measure the quality of internet-available information and consists of 36 items that are divided into 3 subsections. In addition, 5 guideline recommendations per analyzed condition were rephrased as questions and input to ChatGPT, and agreement between the guidelines and the AI answer was measured by 2 authors independently. All queries were repeated 3 times to measure the internal consistency of ChatGPT.
resultsFive conditions were identified (gallstone disease, pancreatitis, liver cirrhosis, pancreatic cancer, and hepatocellular carcinoma). The median EQIP score across all conditions was 16 (IQR 14.5-18) for the total of 36 items. Divided by subsection, median scores for content, identification, and structure data were 10 (IQR 9.5-12.5), 1 (IQR 1-1), and 4 (IQR 4-5), respectively. Agreement between guideline recommendations and answers provided by ChatGPT was 60% (15/25). Interrater agreement as measured by the Fleiss κ was 0.78 (P<.001), indicating substantial agreement. Internal consistency of the answers provided by ChatGPT was 100%.
conclusionsChatGPT provides medical information of comparable quality to available static internet information. Although currently of limited quality, large language models could become the future standard for patients and health care professionals to gather medical information.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.