ArticlePloS one2023
ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center.
Article in PloS one, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 20 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
20 citing papers in PubMed, 1 synthesis or guideline pooled it.
- A systematic review on the generative AI applications in human medical genetics.Frontiers in genetics · 2025Pooled it
- Evaluating AI-based comprehensive clinical decision support for sepsis and ARDS: protocol for a Clinician Turing Test.BMJ open · 2025Article
- Comparison of two different artificial intelligence chatbots that provide information to patients and parents about primary tooth pulpotomy treatments.BMC oral health · 2025Article
- Generative AI Chatbot for Diabetes Management: Formative 2-Part Qualitative Study Using DTalksBot Involving Patients and Clinicians.JMIR formative research · 2025Article
- ChatGPT-4o as a digital health tool for diabetes technology education: insights on reliability, quality, and readability.Endocrine · 2025Article
- Evaluation of the accuracy of ChatGPT-4 and Gemini's responses to the World Dental Federation's frequently asked questions on oral health.BMC oral health · 2025Article
- Applications of large language models in cardiovascular disease: a systematic review.European heart journal. Digital health · 2025Review
- Feasibility of Using ChatGPT to Generate Exposure Hierarchies for Treating Obsessive-Compulsive Disorder.Behavior therapy · 2025Article
- Large Language Model Architectures in Health Care: Scoping Review of Research Perspectives.Journal of medical Internet research · 2025Article
- Article
- Prospects and perils of ChatGPT in diabetes.World journal of diabetes · 2025Article
- Trust in large language model-based solutions in healthcare among people with and without diabetes: a cross-sectional survey from the Health in Central Denmark cohort.BMJ digital health & AI · 2025Article
- Enhancement of the Performance of Large Language Models in Diabetes Education through Retrieval-Augmented Generation: Comparative Study.Journal of medical Internet research · 2024Article
- Response to Generative AI in Dental Licensing Examinations: Comment.International dental journal · 2024Article
- Article
- MED-ChatGPT CoPilot: a ChatGPT medical assistant for case mining and adjunctive therapy.Frontiers in medicine · 2024Article
- ChatGPT: promise and challenges for deployment in low- and middle-income countries.The Lancet regional health. Western Pacific · 2023Review
- Article
- Utilization of ChatGPT for Rhinology Patient Education: Limitations in a Surgical Sub-Specialty.OTO openArticle
- Artificial Intelligence in Speech and Language Therapy: A Qualitative Comparative Analysis of Clinical Applications and Outcomes.Inquiry : a journal of medical care organization, provision and financingArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users' 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.