ArticleJournal of the American Medical Informatics Association : JAMIA2024
Large language models facilitate the generation of electronic health record phenotyping algorithms.
Article in Journal of the American Medical Informatics Association : JAMIA, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 27 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
27 citing papers in PubMed.
- Natural Language Processing Algorithm Accurately Classifies Diverticulitis-Related Complications and Predicts Long-Term Outcomes.Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association · 2026Article
- From study design to executable code: automating target trial emulation with large language models.JAMIA open · 2026Article
- Spatial and telehealth accessibility to eating disorder treatment in the United States: evidence from registry and LLM-augmented data.International journal of health geographics · 2026Article
- Evaluating Large Language Models for Translating Multimodal Phenotype Documentations into Executable EHR Phenotyping Algorithms.Research square · 2026Article
- Evaluating Large Language Models for Translating Multimodal Phenotype Documentations into Executable EHR Phenotyping Algorithms.medRxiv : the preprint server for health sciences · 2026Article
- Development and validation of a multi-institutional electronic juvenile idiopathic arthritis phenotype.Seminars in arthritis and rheumatism · 2026Article
- Applications of Large Language Models in Medical Research: From Systematic Reviews to Clinical Studies.Bioengineering (Basel, Switzerland) · 2026Review
- Improving classification of myocardial infarction with machine learning in a diverse population.American journal of epidemiology · 2026Article
- SHREC: A framework for advancing next-generation computational phenotyping with large language models.PLOS digital health · 2026Article
- Phecoder: semantic retrieval for auditing and expanding ICD-based phenotypes in EHR biobanks.medRxiv : the preprint server for health sciences · 2026Article
- Signal Fidelity Index-aware calibration for addressing distributional shift in predictive modeling across heterogeneous real-world data.Scientific reports · 2025Article
- Zero-Shot Extraction of Seizure Outcomes from Clinical Notes Using Generative Pretrained Transformers.Journal of healthcare informatics research · 2025Article
- Improving automated deep phenotyping through large language models using retrieval-augmented generation.Genome medicine · 2025Article
- SHREC: A Framework for Advancing Next-Generation Computational Phenotyping with Large Language Models.ArXiv · 2025Article
- Iterative Learning of Computable Phenotypes for Treatment Resistant Hypertension using Large Language Models.Proceedings of machine learning research · 2025Article
- Enhancing EHR-based pancreatic cancer prediction with LLM-derived embeddings.NPJ digital medicine · 2025Article
- A GPT-4o-powered framework for identifying cognitive impairment stages in electronic health records.NPJ digital medicine · 2025Article
- Beyond Phecodes: leveraging PheMAP to identify patients lacking diagnosis codes in electronic health records.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Large language models are less effective at clinical prediction tasks than locally trained machine learning models.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Prospects and perils of ChatGPT in diabetes.World journal of diabetes · 2025Article
Corrections and comments
- Update of
Authors and funding
13 authors.
Funding
Abstract
objectivesPhenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts. MATERIALS AND
methodsWe prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.
resultsGPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).
conclusionGPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.