ArticleIEEE journal of biomedical and health informatics2021
Limitations of Transformers on Clinical Text Classification.
Article in IEEE journal of biomedical and health informatics, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 45 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
45 citing papers in PubMed.
- Benchmarking GPT-5, LLaMA, and Mistral for Clinical Named Entity Recognition in Ophthalmology Progress Notes.Translational vision science & technology · 2026Article
- Synthetic artificial intelligence in cardiology: from generative models to clinical applications.European heart journal open · 2026Review
- Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models: Quantitative Study Using Large Language Models.Journal of medical Internet research · 2025Article
- Advances in Artificial Intelligence (AI) Models and Generative Algorithms Represent a New Paradigm for Genomics Research.International journal of molecular sciences · 2025Review
- Leveraging BERT for embedding ICD codes from large scale cardiovascular EMR data to understand patient diagnostic patterns.BMC medical informatics and decision making · 2025Article
- Language Models for Multilabel Document Classification of Surgical Concepts in Exploratory Laparotomy Operative Notes: Algorithm Development Study.JMIR medical informatics · 2025Article
- Enhanced effective convolutional attention network with squeeze-and-excitation inception module for multi-label clinical document classification.Scientific reports · 2025Article
- MISTIC: a novel approach for metastasis classification in Italian electronic health records using transformers.BMC medical informatics and decision making · 2025Article
- Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review.JMIR cancer · 2025Article
- Microblog discourse analysis for parenting style assessment.Frontiers in public health · 2025Article
- Large language models in cancer: potentials, risks, and safeguards.BJR artificial intelligence · 2025Review
- Multimodal BEHRT: transformers for multimodal electronic health records to predict breast cancer prognosis.Frontiers in oncology · 2025Article
- Data transformation of unstructured electroencephalography reports by natural language processing: improving data usability for large-scale epilepsy studies.Frontiers in neurology · 2025Article
- A survey of NLP methods for oncology in the past decade with a focus on cancer registry applications.Artificial intelligence review · 2025Article
- Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset.IEEE journal of translational engineering in health and medicine · 2025Article
- Task-Specific Transformer-Based Language Models in Health Care: Scoping Review.JMIR medical informatics · 2024Article
- Machine learning and deep learning tools for the automated capture of cancer surveillance data.Journal of the National Cancer Institute. Monographs · 2024Article
- Deep-GenMut: Automated genetic mutation classification in oncology: A deep learning comparative study.Heliyon · 2024Article
- DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era.Research square · 2024Article
- Applications of Large Language Models in Pathology.Bioengineering (Basel, Switzerland) · 2024Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors.
Funding
Abstract
Bidirectional Encoder Representations from Transformers (BERT) and BERT-based approaches are the current state-of-the-art in many natural language processing (NLP) tasks; however, their application to document classification on long clinical texts is limited. In this work, we introduce four methods to scale BERT, which by default can only handle input sequences up to approximately 400 words long, to perform document classification on clinical texts several thousand words long. We compare these methods against two much simpler architectures - a word-level convolutional neural network and a hierarchical self-attention network - and show that BERT often cannot beat these simpler baselines when classifying MIMIC-III discharge summaries and SEER cancer pathology reports. In our analysis, we show that two key components of BERT - pretraining and WordPiece tokenization - may actually be inhibiting BERT's performance on clinical text classification tasks where the input document is several thousand words long and where correctly identifying labels may depend more on identifying a few key words or phrases rather than understanding the contextual meaning of sequences of text.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.