ArticleJournal of clinical medicine2023
Examining the Validity of ChatGPT in Identifying Relevant Nephrology Literature: Findings and Implications.
Article in Journal of clinical medicine, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 26 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
26 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review.Journal of the American Medical Informatics Association : JAMIA · 2025Pooled it
- Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.JAMA · 2025Pooled it
- Comparing AI chat tools and PubMed for medical literature searches.Journal of the Medical Library Association : JMLA · 2026Article
- From assistant to collaborator: A systematic review of the evolution of artificial intelligence in end-stage renal disease care and management.PLOS digital health · 2026Article
- A pilot study on the impact of large language model assistance on the evaluation of complex medical living kidney donor candidates.Clinical transplantation and research · 2026Article
- A reproducible statistical evaluation framework for large-sample assessment of AI-generated medical references: cross-platform application of the reference hallucination score.BMC medical research methodology · 2026Article
- Evaluating retrieval augmented generation and ChatGPT's accuracy on orthopaedic examination assessment questions.Annals of joint · 2025Article
- Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis.Journal of healthcare informatics research · 2024Article
- Beyond Multiple-Choice Accuracy: Real-World Challenges of Implementing Large Language Models in Healthcare.ArXiv · 2024Article
- Navigating Nephrology's Decline Through a GPT-4 Analysis of Internal Medicine Specialties in the United States: Qualitative Study.JMIR medical education · 2024Article
- AI-Driven Patient Education in Chronic Kidney Disease: Evaluating Chatbot Responses against Clinical Guidelines.Diseases (Basel, Switzerland) · 2024Article
- The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland.Clinical kidney journal · 2024Article
- Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study.JMIR medical informatics · 2024Article
- Readability analysis of ChatGPT's responses on lung cancer.Scientific reports · 2024Article
- How to improve ChatGPT performance for nephrologists: a technique guide.Journal of nephrology · 2024Article
- Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: Advancing Practical Applications.Medicina (Kaunas, Lithuania) · 2024Review
- Chain of Thought Utilization in Large Language Models and Application in Nephrology.Medicina (Kaunas, Lithuania) · 2024Review
- ChatGPT's competence in addressing urolithiasis: myth or reality?International urology and nephrology · 2024Article
- Performance of ChatGPT on Nephrology Test Questions.Clinical journal of the American Society of Nephrology : CJASN · 2024Article
- Ethical Dilemmas in Using AI for Academic Writing and an Example Framework for Peer Review in Nephrology Academia: A Narrative Review.Clinics and practice · 2023Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Literature reviews are valuable for summarizing and evaluating the available evidence in various medical fields, including nephrology. However, identifying and exploring the potential sources requires focus and time devoted to literature searching for clinicians and researchers. ChatGPT is a novel artificial intelligence (AI) large language model (LLM) renowned for its exceptional ability to generate human-like responses across various tasks. However, whether ChatGPT can effectively assist medical professionals in identifying relevant literature is unclear. Therefore, this study aimed to assess the effectiveness of ChatGPT in identifying references to literature reviews in nephrology. We keyed the prompt "Please provide the references in Vancouver style and their links in recent literature on… name of the topic" into ChatGPT-3.5 (03/23 Version). We selected all the results provided by ChatGPT and assessed them for existence, relevance, and author/link correctness. We recorded each resource's citations, authors, title, journal name, publication year, digital object identifier (DOI), and link. The relevance and correctness of each resource were verified by searching on Google Scholar. Of the total 610 references in the nephrology literature, only 378 (62%) of the references provided by ChatGPT existed, while 31% were fabricated, and 7% of citations were incomplete references. Notably, only 122 (20%) of references were authentic. Additionally, 256 (68%) of the links in the references were found to be incorrect, and the DOI was inaccurate in 206 (54%) of the references. Moreover, among those with a link provided, the link was correct in only 20% of cases, and 3% of the references were irrelevant. Notably, an analysis of specific topics in electrolyte, hemodialysis, and kidney stones found that >60% of the references were inaccurate or misleading, with less reliable authorship and links provided by ChatGPT. Based on our findings, the use of ChatGPT as a sole resource for identifying references to literature reviews in nephrology is not recommended. Future studies could explore ways to improve AI language models' performance in identifying relevant nephrology literature.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.