ArticleBMJ health & care informatics2025
Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation.
Article in BMJ health & care informatics, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 43 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
43 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Research on patient-facing chatbots based on large language models in the care of older people: a living systematic review.European geriatric medicine · 2026Pooled it
- Article
- Automatic Conversion of NICE Guidelines to an Executable Computational Model Using Large Language Models.Learning health systems · 2026Article
- Computational Approaches to Penicillin Allergy De-labelling: A Clinical Review.Clinical reviews in allergy & immunology · 2026Review
- Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease-Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study.JMIR formative research · 2026Article
- Article
- Large Language Model-Based Clinical Decision Support for Antibiotic Selection and Dose Recommendation in Hospitalized Patients With Pneumonia: Multicenter Retrospective Study.JMIR medical informatics · 2026Article
- Extracting adverse event nature, severity, timelines and resulting interventions from clinical notes of patients receiving CAR-T cell therapy using large language models.PLOS digital health · 2026Article
- Iterative Development of an AI-Assisted Data Extraction Tool for Literature Synthesis in Oncology.Cureus · 2026Article
- AI-Assisted Clinical Data Abstraction From Electronic Health Records: Retrospective Concordance Study.JMIR formative research · 2026Article
- From API to Action: A Multi-Model Comparison of OpenAI, Anthropic, Google, and Meta LLMs for Clinical Trial Data Extraction.Bioengineering (Basel, Switzerland) · 2026Article
- Evaluation Frameworks for Clinical Foundation Models in Specific Tasks of Unstructured Medical Text Analysis: A Scoping Review.Healthcare (Basel, Switzerland) · 2026Review
- Extracting and Classifying Drug Discontinuations From Estonian Electronic Health Records: Development and Validation Study.Journal of medical Internet research · 2026Article
- Open-source large language model-based on-premises pipeline for automated data extraction from unstructured electronic health records: a pilot study.BMJ health & care informatics · 2026Article
- Large language models approach clinician performance in ESC cardiovascular risk stratification: a vignette-based benchmark study.European heart journal. Digital health · 2026Article
- Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing.Journal of the American Medical Informatics Association : JAMIA · 2026Article
- Using large language models to enhance clinically-driven missing data recovery algorithms in electronic health records.JAMIA open · 2026Article
- Spatial and telehealth accessibility to eating disorder treatment in the United States: evidence from registry and LLM-augmented data.International journal of health geographics · 2026Article
- A generative approach for semantic auditing of electronic health records.NPJ digital medicine · 2026Article
- Ant Colony Optimization-Driven Ensemble Learning for Carbon Emission Modelling in Fly Ash-Slag Geopolymer Concrete.Materials (Basel, Switzerland) · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectivesWe aimed to evaluate the performance of multiple large language models (LLMs) in data extraction from unstructured and semi-structured electronic health records.
methods50 synthetic medical notes in English, containing a structured and an unstructured part, were drafted and evaluated by domain experts, and subsequently used for LLM-prompting. 18 LLMs were evaluated against a baseline transformer-based model. Performance assessment comprised four entity extraction and five binary classification tasks with a total of 450 predictions for each LLM. LLM-response consistency assessment was performed over three same-prompt iterations.
resultsClaude 3.0 Opus, Claude 3.0 Sonnet, Claude 2.0, GPT 4, Claude 2.1, Gemini Advanced, PaLM 2 chat-bison and Llama 3-70b exhibited an excellent overall accuracy >0.98 (0.995, 0.988, 0.988, 0.988, 0.986, 0.982, 0.982, and 0.982, respectively), significantly higher than the baseline RoBERTa model (0.742). Claude 2.0, Claude 2.1, Claude 3.0 Opus, PaLM 2 chat-bison, GPT 4, Claude 3.0 Sonnet and Llama 3-70b showed a marginally higher and Gemini Advanced a marginally lower multiple-run consistency than the baseline model RoBERTa (Krippendorff's alpha value 1, 0.998, 0.996, 0.996, 0.992, 0.991, 0.989, 0.988, and 0.985, respectively). DISCUSSION: Claude 3.0 Opus, Claude 3.0 Sonnet, Claude 2.0, GPT 4, Claude 2.1, Gemini Advanced, PaLM 2 chat bison and Llama 3-70b performed the best, exhibiting outstanding performance in both entity extraction and binary classification, with highly consistent responses over multiple same-prompt iterations. Their use could leverage data for research and unburden healthcare professionals. Real-data analyses are warranted to confirm their performance in a real-world setting.
conclusionClaude 3.0 Opus, Claude 3.0 Sonnet, Claude 2.0, GPT 4, Claude 2.1, Gemini Advanced, PaLM 2 chat-bison and Llama 3-70b seem to be able to reliably extract data from unstructured and semi-structured electronic health records. Further analyses using real data are warranted to confirm their performance in a real-world setting.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.