ArticlePLOS digital health2024
Evaluating Large Language Models in extracting cognitive exam dates and scores.
Article in PLOS digital health, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
15 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Generative large language models in the clinical management of Alzheimer's disease and mild cognitive impairment.Neurological sciences : official journal of the Italian Neurological Society and of the Italian Society of Clinical Neurophysiology · 2026Pooled it
- Artificial Intelligence for Alzheimer's Disease Diagnosis: From Traditional Machine Learning to Large Language Models.Biosensors · 2026Review
- GLP-1-based incretin therapy is associated with lower incident Alzheimer's disease and cardiorenal events in adults with documented neuropsychiatric, cognitive, or sensory risk.Biology methods & protocols · 2026Article
- Psychometric applications of generative artificial intelligence: Lifecycle, risks, and research agenda.PLOS mental health · 2026Review
- Enhancing the prediction of hospital discharge disposition with extraction-based language model classification.npj health systems · 2026Article
- Scalable scientific interest profiling using large language models.Journal of biomedical informatics · 2025Article
- Evaluating the diagnostic accuracy of vision language models for neuroradiological image interpretation.NPJ digital medicine · 2025Article
- Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke.Journal of neurointerventional surgery · 2025Article
- Automated structured data extraction from intraoperative echocardiography reports using large language models.British journal of anaesthesia · 2025Article
- AI modeling for outbreak prediction: A graph-neural-network approach for identifying vancomycin-resistant enterococcus carriers.PLOS digital health · 2025Article
- From Text to Data: Automatically Extracting Data From Catheterization Reports Using Generative Artificial Intelligence.Journal of the Society for Cardiovascular Angiography & Interventions · 2025Article
- Article
- Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports.Radiology. Artificial intelligence · 2024Article
- Predicting Risk of Alzheimer's Diseases and Related Dementias with AI Foundation Model on Electronic Health Records.medRxiv : the preprint server for health sciences · 2024Article
- SEETrials: Leveraging large language models for safety and efficacy extraction in oncology clinical trials.Informatics in medicine unlocked · 2024Article
Corrections and comments
- Update of
Authors and funding
32 authors.
Funding
Abstract
Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.