ArticleApplied clinical informatics2025
Extracting International Classification of Diseases Codes from Clinical Documentation Using Large Language Models.
Article in Applied clinical informatics, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 11 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
11 citing papers in PubMed.
- Augmenting structured diagnoses through effective use of pre-trained large language models on clinical notes.JAMIA open · 2026Article
- Machine Learning to Identify Point-of-Care Ultrasound and Evaluate Standardized Documentation: Retrospective Operational Cohort Study.Journal of medical Internet research · 2026Article
- Evaluating real-world deployment of an HL7-CDA-aligned LLM for ICD-10-CM coding.NPJ digital medicine · 2026Article
- Evaluation of Large Language Models for Automated SimpleFoot & ankle orthopaedics · 2026Article
- Generative AI costs in large healthcare systems, an example in revenue cycle.NPJ digital medicine · 2025Article
- Can LLMs effectively assist medical coding? Evaluating GPT performance on DRG and targeted clinical tasks.BMC medical informatics and decision making · 2025Article
- Using Open-Source Large Language Models to Identify Access to Germline Genetic Testing in Veterans With Breast Cancer From Unstructured Text.JCO clinical cancer informatics · 2025Article
- Primer on large language models: an educational overview for intensivists.Critical care (London, England) · 2025Review
- Prediction of Lupus Classification Criteria via Generative AI Medical Record Profiling.Biotech (Basel (Switzerland)) · 2025Article
- Article
- Article
Corrections and comments
- Update of
Authors and funding
17 authors.
Funding
Abstract
backgroundLarge language models (LLMs) have shown promise in various professional fields, including medicine and law. However, their performance in highly specialized tasks, such as extracting ICD-10-CM codes from patient notes, remains underexplored.
objectiveThe primary objective was to evaluate and compare the performance of ICD-10-CM code extraction by different LLMs with that of human coder.
methodsWe evaluated performance of six LLMs (GPT-3.5, GPT-4, Claude 2.1, Claude 3, Gemini Advanced, and Llama 2-70b) in extracting ICD-10-CM codes against human coder. We used deidentified inpatient notes of authentic patient cases from American Health Information Management Association Vlab for this study. We calculated percent agreement and Cohen's kappa values to assess the agreement between LLMs and human coder. We then identified reasons for discrepancies in code extraction by LLMs in a 10% random subset.
resultsAmong 50 inpatient notes, human coder extracted 165 unique ICD-10-CM codes. LLMs extracted significantly higher number of unique ICD-10-CM codes than human coder, with Llama 2-70b extracting most (658) and Gemini Advanced the least (221). GPT-4 achieved highest percent agreement with human coder at 15.2%, followed by Claude 3 (12.7%) and GPT-3.5 (12.4%). Cohen's kappa values indicated minimal to no agreement, ranging from -0.02 to 0.01. When focusing on primary diagnosis, Claude 3 achieved highest percent agreement (26%) and kappa value (0.25). Reasons for discrepancies in extraction of codes varied among LLMs and included extraction of codes for diagnoses not confirmed by providers (60% with GPT-4), extraction of nonspecific codes (25% with GPT-3.5), extraction of codes for signs and symptoms despite presence of more specific diagnosis (22% with Claude 2.1), and hallucinations (35% with Claude 2.1).
conclusionCurrent LLMs have poor performance in extraction of ICD-10-CM codes from inpatient notes when compared against the human coder.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.