ArticleJournal of medical Internet research2024
Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study.
Article in Journal of medical Internet research, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
21 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis.Frontiers in research metrics and analytics · 2026Pooled it
- Accuracy and Error Patterns of References Generated by Large Language Models in Endodontics: The Role of Prompt Design and Model Selection.Medical science monitor : international medical journal of experimental and clinical research · 2026Article
- Teaching the Atomic Sentence Method for Source-Verified, AI-Assisted Literature Synthesis to Clinical Health Care Professionals: Single-Cohort Feasibility and Acceptability Study.JMIR formative research · 2026Article
- Leveraging generative artificial intelligence errors to teach appropriate citation usage.Journal of microbiology & biology education · 2026Article
- The role of large language models in the writing of surgical reviews: fact or fantasy?Langenbeck's archives of surgery · 2026Article
- A large language model pipeline for automated citation quality scoring across engineering journal quartiles with expert validation.Scientific reports · 2026Article
- Evaluating Source-Based Large Language Models for Preclinical Dermatology Education: Comparative Study.JMIR formative research · 2026Article
- Comparing the performance of four mainstream large language models on medical literature review generation: a human expert evaluation in SMILE surgery.Graefe's archive for clinical and experimental ophthalmology = Albrecht von Graefes Archiv fur klinische und experimentelle Ophthalmologie · 2026Article
- Evaluating the reference accuracy of large language models in radiology: a comparative study across subspecialties.Diagnostic and interventional radiology (Ankara, Turkey) · 2026Observational
- Citation integrity in the AI era: a focused evidence map and layered verification framework for gastroenterology and hepatology publishing.Gastroenterology report · 2026Article
- Autonomous artificial intelligence, scientific research, and human values.AI and ethics · 2026Article
- Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?Medical science monitor : international medical journal of experimental and clinical research · 2026Article
- Assessing the Quality of AI Responses to Patient Concerns About Axial Spondyloarthritis: Delphi-Based Evaluation.JMIR AI · 2026Article
- Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study.JMIR mental health · 2025Article
- Using Artificial Intelligence for Scholarly Writing.The American journal of nursing · 2025Article
- Comparative performance of neurosurgery-specific, peer-reviewed versus general AI chatbots in bilingual board examinations: evaluating accuracy, consistency, and error minimization strategies.Acta neurochirurgica · 2025Article
- False authorship: an explorative case study around an AI-generated article published under my name.Research integrity and peer review · 2025Article
- Cross sectional pilot study on clinical review generation using large language models.NPJ digital medicine · 2025Article
- Is your curriculum GenAI-proof? A method for GenAI impact assessment and a case study.MedEdPublish (2016) · 2025Article
- Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses.Journal of medical Internet research · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge language models (LLMs) have gained prominence since the release of ChatGPT in late 2022.
objectiveThe aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities.
methodsTwo researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines.
resultsTen topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy.
conclusionsChatGPT's performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.