ArticleClinical journal of the American Society of Nephrology : CJASN2024
Performance of ChatGPT on Nephrology Test Questions.
Article in Clinical journal of the American Society of Nephrology : CJASN, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 49 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
49 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Digital pathology and artificial intelligence in renal cell carcinoma focusing on feature extraction: a literature review.Frontiers in oncology · 2025Pooled it
- Performance of Reasoning and Comparator Large Language Models on Nephrology Multiple-choice Questions.JMA journal · 2026Article
- Randomized Controlled Trial of Large Language Model-Assisted Diagnostic Accuracy in Nephrology.Kidney international reports · 2026Article
- ChatGPT Versus DeepSeek: Assessing Artificial Intelligence Performance on Radiation Oncology Examination Questions.Advances in radiation oncology · 2026Article
- Performance of Large Language Models in Analyzing Common Hypertension Scenarios.Hypertension (Dallas, Tex. : 1979) · 2026Article
- Token-splitting improves GPT-4.1 performance on plastic surgery exams: implications for AI-Assisted medical education.Medical education online · 2025Article
- Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study.JMIR medical education · 2025Article
- Accuracy, Clarity, and Comprehensiveness of ChatGPT Outputs for Commonly Asked Questions About Living Kidney Donation.Clinical transplantation · 2025Article
- Comparative Analysis of Large Language Model and Physician-Generated Responses in Bariatric Patient Inquiries: Assessing the Accuracy and Patient Satisfaction.Obesity surgery · 2025Article
- The application of problem-based learning (PBL) guided by ChatGPT in clinical education in the Department of Nephrology.BMC medical education · 2025Article
- GPT-4's performance in supporting physician decision-making in nephrology multiple-choice questions.Scientific reports · 2025Article
- Evaluating AI performance in nephrology triage and subspecialty referrals.Scientific reports · 2025Article
- Exploring the performance of large language models on hepatitis B infection-related questions: A comparative study.World journal of gastroenterology · 2025Article
- Performance of o1 pro and GPT-4 in Self-Assessment Questions for Nephrology Board Renewal.Frontiers in medicine · 2025Article
- Enhancing Patient Comprehension of Glomerular Disease Treatments Using ChatGPT.Healthcare (Basel, Switzerland) · 2024Article
- Identification of kidney-related medications using AI from self-captured pill images.Renal failure · 2024Article
- Perspectives on AI-based recommendations for mask-wearing and COVID-19 vaccination for transplant recipients in the post-COVID-19 era.Renal failure · 2024Article
- Assessing the Performance of Chatbots on the Taiwan Psychiatry Licensing Examination Using the Rasch Model.Healthcare (Basel, Switzerland) · 2024Article
- Navigating Nephrology's Decline Through a GPT-4 Analysis of Internal Medicine Specialties in the United States: Qualitative Study.JMIR medical education · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundChatGPT is a novel tool that allows people to engage in conversations with an advanced machine learning model. ChatGPT's performance in the US Medical Licensing Examination is comparable with a successful candidate's performance. However, its performance in the nephrology field remains undetermined. This study assessed ChatGPT's capabilities in answering nephrology test questions.
methodsQuestions sourced from Nephrology Self-Assessment Program and Kidney Self-Assessment Program were used, each with multiple-choice single-answer questions. Questions containing visual elements were excluded. Each question bank was run twice using GPT-3.5 and GPT-4. Total accuracy rate, defined as the percentage of correct answers obtained by ChatGPT in either the first or second run, and the total concordance, defined as the percentage of identical answers provided by ChatGPT during both runs, regardless of their correctness, were used to assess its performance.
resultsA comprehensive assessment was conducted on a set of 975 questions, comprising 508 questions from Nephrology Self-Assessment Program and 467 from Kidney Self-Assessment Program. GPT-3.5 resulted in a total accuracy rate of 51%. Notably, the employment of Nephrology Self-Assessment Program yielded a higher accuracy rate compared with Kidney Self-Assessment Program (58% versus 44%; P < 0.001). The total concordance rate across all questions was 78%, with correct answers exhibiting a higher concordance rate (84%) compared with incorrect answers (73%) ( P < 0.001). When examining various nephrology subfields, the total accuracy rates were relatively lower in electrolyte and acid-base disorder, glomerular disease, and kidney-related bone and stone disorders. The total accuracy rate of GPT-4's response was 74%, higher than GPT-3.5 ( P < 0.001) but remained below the passing threshold and average scores of nephrology examinees (77%).
conclusionsChatGPT exhibited limitations regarding accuracy and repeatability when addressing nephrology-related questions. Variations in performance were evident across various subfields.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.