ArticleProceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence2024
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records.
Article in Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 34 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
34 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.JAMA · 2025Pooled it
- Automated evaluation can distinguish the good and bad AI responses to patient questions about hospitalization.NPJ digital medicine · 2026Article
- Article
- Structured taxonomy and framework for developing medical benchmark in large language models derived from scoping review.NPJ digital medicine · 2026Article
- Holistic evaluation of large language models for medical tasks with MedHELM.Nature medicine · 2026Article
- A personal health large language model for sleep and fitness coaching.Nature medicine · 2025Article
- TIMER: temporal instruction modeling and evaluation for longitudinal clinical records.NPJ digital medicine · 2025Article
- Iterative refinement and goal articulation to optimize large language models for clinical information extraction.NPJ digital medicine · 2025Article
- Article
- Benchmarking large language models for biomedical natural language processing applications and recommendations.Nature communications · 2025Article
- A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings.Nature communications · 2025Article
- Prompts to Table: Specification and Iterative Refinement for Clinical Information Extraction with Large Language Models.medRxiv : the preprint server for health sciences · 2025Article
- Large language model agents can use tools to perform clinical calculations.NPJ digital medicine · 2025Article
- A dataset and benchmark for hospital course summarization with adapted large language models.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Foundation Models in Radiology: What, How, Why, and Why Not.Radiology · 2025Review
- Towards evaluating and building versatile large language models for medicine.NPJ digital medicine · 2025Article
- Clinical entity augmented retrieval for clinical information extraction.NPJ digital medicine · 2025Article
- Large language models and rheumatology: are we there yet?Rheumatology advances in practice · 2025Review
- Frequent use of online medical records: analysis of influence factors based on structural equation modeling.Frontiers in public health · 2025Article
- Biomedical Visual Instruction Tuning with Clinician Preference Alignment.Advances in neural information processing systems · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
30 authors.
Funding
Abstract
The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. MedAlign is provided under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.