ArticleJMIR medical education2023
Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study.
Article in JMIR medical education, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 29 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
29 citing papers in PubMed.
- Impact of Artificial Intelligence-Generated Content Labels On Perceived Accuracy, Message Credibility, and Sharing Intentions for Misinformation: Web-Based, Randomized, Controlled Experiment.JMIR formative research · 2024Trial
- A collaborative large language model for drug analysis.Nature biomedical engineering · 2026Article
- Large language models for bioinformatics.Quantitative biology (Beijing, China) · 2026Review
- Generative AI in academia: Efficiency versus scholarship.The Journal of physiology · 2026Article
- Classifying human vs. AI text with machine learning and explainable transformer models.Scientific reports · 2025Article
- Can we trust academic AI detective? Accuracy and limitations of AI-output detectors.Acta neurochirurgica · 2025Article
- A visualization system for intelligent diagnosis and statistical analysis of oral diseases based on panoramic radiography.Scientific reports · 2025Article
- Feasibility study of automatic radiotherapy treatment planning for cervical cancer using a large language model.Radiation oncology (London, England) · 2025Article
- From manual clinical criteria to machine learning algorithms: Comparing outcome endpoints derived from diverse electronic health record data modalities.PLOS digital health · 2025Article
- Evaluating the Potential of Large Language Models for Vestibular Rehabilitation Education: A Comparison of ChatGPT, Google Gemini, and Clinicians.Physical therapy · 2025Article
- Detecting Artificial Intelligence-Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study.JMIR medical education · 2025Article
- Optimizing academic engagement and mental health through AI: an experimental study on LLM integration in higher education.Frontiers in psychology · 2025Article
- AD-AutoGPT: An autonomous GPT for Alzheimer's disease infodemiology.PLOS global public health · 2025Article
- Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis.Journal of healthcare informatics research · 2024Article
- Conceptual review of outcome metrics and measures used in clinical evaluation of artificial intelligence in radiology.La Radiologia medica · 2024Review
- SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization.Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing · 2024Article
- Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study.JMIR AI · 2024Article
- Unveiling ChatGPT text using writing style.Heliyon · 2024Article
- Constructing synthetic datasets with generative artificial intelligence to train large language models to classify acute renal failure from clinical notes.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis.JMIR formative research · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge language models, such as ChatGPT, are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the internet. However, medical texts, such as clinical notes and diagnoses, require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to health care and the general public.
objectiveThis study is among the first on responsible artificial intelligence-generated content in medicine. We focus on analyzing the differences between medical texts written by human experts and those generated by ChatGPT and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT.
methodsWe first constructed a suite of data sets containing medical texts written by human experts and generated by ChatGPT. We analyzed the linguistic features of these 2 types of content and uncovered differences in vocabulary, parts-of-speech, dependency, sentiment, perplexity, and other aspects. Finally, we designed and implemented machine learning methods to detect medical text generated by ChatGPT. The data and code used in this paper are published on GitHub.
resultsMedical texts written by humans were more concrete, more diverse, and typically contained more useful information, while medical texts generated by ChatGPT paid more attention to fluency and logic and usually expressed general terminologies rather than effective information specific to the context of the problem. A bidirectional encoder representations from transformers-based model effectively detected medical texts generated by ChatGPT, and the F
conclusionsAlthough text generated by ChatGPT is grammatically perfect and human-like, the linguistic characteristics of generated medical texts were different from those written by human experts. Medical text generated by ChatGPT could be effectively detected by the proposed machine learning algorithms. This study provides a pathway toward trustworthy and accountable use of large language models in medicine.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.