ArticleAnnals of surgical oncology2024
Performance of Artificial Intelligence Content Detectors Using Human and Artificial Intelligence-Generated Scientific Writing.
Article in Annals of surgical oncology, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 10 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
10 citing papers in PubMed.
- Artificial Intelligence in Veterinary Peer Review: Advancing Rigor or Raising New Concerns?Veterinary medicine and science · 2026Article
- AI text detection in dentistry: a comparative analysis across generative models.Research integrity and peer review · 2026Article
- Artificial Intelligence in Scientific Writing: Balancing Innovation and Efficiency with Integrity: Perspectives and Position Statements of Global Andrology Forum Expert Group.The world journal of men's health · 2026Article
- ChatGPT in Academic Writing: A Scientometric Analysis of Literature Published Between 2022 and 2023.Journal of empirical research on human research ethics : JERHRE · 2025Article
- Letter to the Editor: The Fate of Individual Tone in the Age of AI Writing.Annals of surgical oncology · 2025Article
- Do I Write Like Artificial Intelligence?Annals of surgical oncology · 2025Article
- How do oncology journals approach plagiarism? A website review.Research integrity and peer review · 2025Article
- Article
- AI detectors are poor western blot classifiers: a study of accuracy and predictive values.PeerJ · 2025Article
- Generative artificial intelligence in ophthalmology research writing: A comprehensive review of applications, detection tools, and ethical considerations.Taiwan journal of ophthalmologyReview
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundFew studies have examined the performance of artificial intelligence (AI) content detection in scientific writing. This study evaluates the performance of publicly available AI content detectors when applied to both human-written and AI-generated scientific articles.
methodsArticles published in Annals of Surgical Oncology (ASO) during the year 2022, as well as AI-generated articles using OpenAI's ChatGPT, were analyzed by three AI content detectors to assess the probability of AI-generated content. Full manuscripts and their individual sections were evaluated. Group comparisons and trend analyses were conducted by using ANOVA and linear regression. Classification performance was determined using area under the curve (AUC).
resultsA total of 449 original articles met inclusion criteria and were evaluated to determine the likelihood of being generated by AI. Each detector also evaluated 47 AI-generated articles by using titles from ASO articles. Human-written articles had an average probability of being AI-generated of 9.4% with significant differences between the detectors. Only two (0.4%) human-written manuscripts were detected as having a 0% probability of being AI-generated by all three detectors. Completely AI-generated articles were evaluated to have a higher average probability of being AI-generated (43.5%) with a range from 12.0 to 99.9%.
conclusionsThis study demonstrates differences in the performance of various AI content detectors with the potential to label human-written articles as AI-generated. Any effort toward implementing AI detectors must include a strategy for continuous evaluation and validation as AI models and detectors rapidly evolve.
Indexed as
Identifiers
38909113What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.