Evidence map›Paper›PMID 41962200›Full record

ArticleEBioMedicine2026

Assessing the quality of AI-generated and physician-written discharge summaries: evaluation of an EHR-integrated tool in a Dutch academic hospital.

Tarannom Mehri, Tun Nadalini, Anne H Hoekman, Tom P van der Laan, Katerina Kagialari, Robert K Wagner, Job N Doornberg, Rosanne C Schoonbeek, Charlotte M H H T Bootsma-Robroeks, Applied Artificial Intelligence in Healthcare Consortium

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in EBioMedicine, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 2 papers.

0numbers the graph read from it
0cells of the map it votes in
2citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

2 citing papers in PubMed.

  1. Article
  2. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

10 authors.

Tarannom MehriDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands. Electronic address: t.mehri@umcg.nl.
Tun NadaliniDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands.
Anne H HoekmanDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands.
Tom P van der LaanDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands; Department of Otolaryngology, Head and Neck Surgery, University Medical Centre, Groningen, the Netherlands.
Katerina KagialariDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands.
Robert K WagnerDepartment of Orthopaedic Surgery, Massachusetts General Hospital, Boston, MA, USA; Harvard Medical School Orthopaedic Trauma Initiative, Boston, MA, USA; Department of Orthopaedic Surgery and Sports Medicine, Amsterdam UMC Location University of Amsterdam, Meibergdreef 9, Amsterdam, the Netherlands.
Job N DoornbergDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands; Department of Trauma Surgery / Orthopaedics, University Medical Centre, Groningen, the Netherlands; Harvard Medical School Orthopaedic Trauma Initiative, Boston, MA, USA; Department of Orthopaedic Trauma, Flinders University Medical Centre, Adelaide, Australia.
Rosanne C SchoonbeekDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands; Department of Otolaryngology, Head and Neck Surgery, University Medical Centre, Groningen, the Netherlands.
Charlotte M H H T Bootsma-RobroeksDepartment of Health Information Office, Information Management Healthcare, University Medical Centre, Groningen, the Netherlands; Department of Paediatrics, Paediatrics Nephrology, Beatrix Children's Hospital, University Medical Centre, Groningen, the Netherlands.
Applied Artificial Intelligence in Healthcare Consortium

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

backgroundLarge language models (LLMs) offer potential to reduce administrative burden in clinical care by generating discharge summaries. Most prior evaluations have been limited to drafts, small cohorts, or non-integrated settings. Robust validation of fully automated, EHR-integrated systems in real-world practice is lacking.

methodsThis study was conducted in April 2025 at a Dutch academic hospital. A total of 292 paired discharge summaries from multiple departments were evaluated, each consisting of a physician-written and an LLM-generated version. Summaries were independently assessed by eight blinded clinicians using a 5-point Likert scale across completeness, correctness, and conciseness. Trustworthiness was also scored. Domain and total scores were compared with Wilcoxon signed-rank tests, and interrater reliability was quantified using Gwet's AC2.

findingsLLM-generated summaries had lower completeness (4.50 (4.00-5.00) vs 5.00 (4.50-5.00); p < 0.001), similar correctness (5.00 (4.50-5.00) vs 5.00 (4.63-5.00); p = 0.14), and greater conciseness (5.00 (4.50-5.00) vs 4.50 (4.00-5.00); p < 0.001) compared with physician-written summaries. Total scores did not differ (14.00 (13.00-15.00) vs 14.00 (13.00-15.00); p = 0.34). Physician-written summaries were trusted by both reviewers in 279 (95.5%) cases, whereas LLM-generated summaries were trusted in 249 (85.3%) cases, partially trusted in 34 (11.6%), and rejected in 9 (3.1%). Interrater agreement for total scores was high (AC2 0.87, 95% CI 0.83-0.90 for LLM; 0.85, 95% CI 0.81-0.89 for physician).

interpretationDischarge summaries generated by an EHR-integrated LLM achieved quality ratings comparable to physician-written documents across multiple specialties, with no difference in total scores. Unlike earlier pilot work, this study demonstrates real-world feasibility of automated LLM use in clinical workflows at scale. With appropriate oversight and specialty-specific refinement, such systems could substantially reduce documentation burden while maintaining discharge summary quality.

fundingThis research did not receive a specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Indexed as

Artificial IntelligenceElectronic Health RecordsPatient DischargePatient Discharge SummariesAcademic Medical CentersHumansLarge Language ModelsNetherlandsPhysiciansReproducibility of ResultsClinical documentationDischarge summariesElectronic health recordsHealth informaticsLarge language modelsValidation study

Identifiers

PMID41962200
PMCPMC13091380

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.