ArticleJournal of the American Medical Informatics Association : JAMIA2023
A comparative study of pretrained language models for long clinical text.
Article in Journal of the American Medical Informatics Association : JAMIA, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 56 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
56 citing papers in PubMed.
- AI-assisted post-mortem classification of vehicle trauma in free-roaming cats using transformer-based large language models.The veterinary quarterly · 2026Article
- Fine-tuning and evaluating large language models for patient safety tasks: classification of contributing factors in incident reports.Journal of the American Medical Informatics Association : JAMIA · 2026Article
- Microbiome and aging: Trajectories of microbiome age across human ecosystems and their systemic effects.iMeta · 2026Review
- Clinical drug report generation using multi-phase prompt large language models.Scientific reports · 2026Article
- Automated identification of incidentalomas requiring follow-up: A multi-anatomy evaluation of LLM-based and supervised approaches.Journal of biomedical informatics · 2026Article
- A pretrained biomedical large language model for Persian biomedical text mining.Scientific reports · 2026Article
- An Artificial Intelligence-Based Detection of Comorbid Depression, Anxiety, and Substance Use Disorder in Korean Counseling Dialogues Using an Explainable Hierarchical Attention Network with Shapley Additive Explanations.Diagnostics (Basel, Switzerland) · 2026Article
- Prognostic data extraction harnessing a privacy-preserving large language model: a clinician-AI collaborative retrospective evaluation in head and neck oncology.NPJ precision oncology · 2026Article
- Combining generative large language models and pretrained language models for predicting acute respiratory distress syndrome progression.Medical & biological engineering & computing · 2026Article
- Scalable Identification of Clinically Relevant Chronic Obstructive Pulmonary Disease Documents in Large-Scale Electronic Health Record Datasets With a Lightweight Natural Language Processing Model: Retrospective Cohort Study.JMIR medical informatics · 2026Article
- Article
- ClinicRealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks.NPJ digital medicine · 2026Article
- Article
- Privacy-Preserving Generation of Structured Lymphoma Progression Reports from Cross-sectional Imaging: A Comparative Analysis of Llama 3.3 and Llama 4.Journal of imaging informatics in medicine · 2026Article
- Temporal Annotation of German Clinical Language in Real and Synthetic Clinical Documents: Corpus Development and Baseline Tagger Validation Study.Journal of medical Internet research · 2026Article
- Review
- Article
- Enhancing cause of death prediction: development and validation of machine learning models using multimodal data across multiple health-care sites.JAMIA open · 2026Article
- Enhancing the prediction of hospital discharge disposition with extraction-based language model classification.npj health systems · 2026Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
Abstract
objectiveClinical knowledge-enriched transformer models (eg, ClinicalBERT) have state-of-the-art results on clinical natural language processing (NLP) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (eg, Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts. MATERIALS AND
methodsInspired by the success of long-sequence transformer models and the fact that clinical notes are mostly long, we introduce 2 domain-enriched language models, Clinical-Longformer and Clinical-BigBird, which are pretrained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks.
resultsThe results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results. DISCUSSION: Our pretrained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https://github.com/luoyuanlab/Clinical-Longformer, and the pretrained models available for public download at: https://huggingface.co/yikuan8/Clinical-Longformer.
conclusionThis study demonstrates that clinical knowledge-enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.