ArticleDiagnostics (Basel, Switzerland)2023
Evaluating the Utility of a Large Language Model in Answering Common Patients' Gastrointestinal Health-Related Questions: Are We There Yet?
Article in Diagnostics (Basel, Switzerland), 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 51 papers, 4 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
51 citing papers in PubMed, 4 syntheses or guidelines pooled it, 94 citations in OpenAlex.
- Large Language Models for Chatbot Health Advice Studies: A Systematic Review.JAMA network open · 2025Pooled it
- Large Language Models in Gastroenterology: Systematic Review.Journal of medical Internet research · 2024Pooled it
- Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review.Journal of medical Internet research · 2024Pooled it
- Emerging applications of NLP and large language models in gastroenterology and hepatology: a systematic review.Frontiers in medicine · 2024Pooled it
- Factors Shaping Trust and Satisfaction With AI Medical Chatbots: A Mixed Methods Vignette Survey of Caregivers Seeking Guidance on Pediatric Infectious Diseases.Journal of medical Internet research · 2026Article
- A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded multi-stakeholder assessment and guideline comparison.Rheumatology international · 2026Observational
- Gut Instincts, Machine Decisions: Evaluating AI Accuracy in the Diagnosis and Treatment of Disorders of Gut-Brain Interaction.Neurogastroenterology and motility · 2026Article
- Evaluation of the accuracy, quality, and readability of large language models on local anesthesia and general anesthesia-sedation in dentistry.BMC anesthesiology · 2026Article
- Comparing AI Chatbots to Live Practitioners of Homeopathy: A Comparative Retrospective Study.Healthcare (Basel, Switzerland) · 2026Article
- Comparative analysis of Chinese large language model performance on atrial fibrillation questions.BMC cardiovascular disorders · 2026Article
- Foundation models: Insights and implications for gastrointestinal cancer.World journal of gastroenterology · 2025Review
- The role of large language models in self-care: a study and benchmark on medicines and supplement guidance accuracy.International journal of clinical pharmacy · 2025Article
- The potential utility of CHATGPT4.0 as an AI assistant in the education and management of patients with Barrett's esophagus.Diseases of the esophagus : official journal of the International Society for Diseases of the Esophagus · 2025Article
- Comparative performance analysis of global and chinese-domain large language models for myopia.Eye (London, England) · 2025Article
- Clinical applications of large language models in medicine and surgery: A scoping review.The Journal of international medical research · 2025Article
- Large language models' capabilities in responding to tuberculosis medical questions: testing ChatGPT, Gemini, and Copilot.Scientific reports · 2025Article
- Article
- Large Language Models in Gastroenterology and Gastrointestinal Surgery: A New Frontier in Patient Communication and Education.Gastroenterology research · 2025Review
- Evaluating the quality of medical content on YouTube using large language models.Scientific reports · 2025Article
- The use of artificial intelligence based chat bots in ophthalmology triage.Eye (London, England) · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors at 2 institutions in 2 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
BACKGROUND AND
aimsPatients frequently have concerns about their disease and find it challenging to obtain accurate Information. OpenAI's ChatGPT chatbot (ChatGPT) is a new large language model developed to provide answers to a wide range of questions in various fields. Our aim is to evaluate the performance of ChatGPT in answering patients' questions regarding gastrointestinal health.
methodsTo evaluate the performance of ChatGPT in answering patients' questions, we used a representative sample of 110 real-life questions. The answers provided by ChatGPT were rated in consensus by three experienced gastroenterologists. The accuracy, clarity, and efficacy of the answers provided by ChatGPT were assessed.
resultsChatGPT was able to provide accurate and clear answers to patients' questions in some cases, but not in others. For questions about treatments, the average accuracy, clarity, and efficacy scores (1 to 5) were 3.9 ± 0.8, 3.9 ± 0.9, and 3.3 ± 0.9, respectively. For symptoms questions, the average accuracy, clarity, and efficacy scores were 3.4 ± 0.8, 3.7 ± 0.7, and 3.2 ± 0.7, respectively. For diagnostic test questions, the average accuracy, clarity, and efficacy scores were 3.7 ± 1.7, 3.7 ± 1.8, and 3.5 ± 1.7, respectively.
conclusionsWhile ChatGPT has potential as a source of information, further development is needed. The quality of information is contingent upon the quality of the online information provided. These findings may be useful for healthcare providers and patients alike in understanding the capabilities and limitations of ChatGPT.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.