Evidence map›Paper›PMID 34823030›Full record

ArticleJournal of biomedical informatics2022

Class imbalance in out-of-distribution datasets: Improving the robustness of the TextCNN for the classification of rare cancer types.

Kevin De Angeli, Shang Gao, Ioana Danciu, Eric B Durbin, Xiao-Cheng Wu, Antoinette Stroup, Jennifer Doherty, Stephen Schwartz, Charles Wiggins, Mark Damesyn and 4 more

Abstract read
In one paragraph

Article in Journal of biomedical informatics, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 18 papers.

0numbers the graph read from it
0cells of the map it votes in
18citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

18 citing papers in PubMed.

  1. Article
  2. Review
  3. Review
  4. Review
  5. Article
  6. Article
  7. Article
  8. Article
  9. Article
  10. Article
  11. The SEER Program's evolution: supporting clinically meaningful population-level research.Journal of the National Cancer Institute. Monographs · 2024
    Article
  12. Article
  13. Time Matters: Examine Temporal Effects on Biomedical Language Models.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2024
    Article
  14. Topological Superconductors from a Materials Perspective.Chemistry of materials : a publication of the American Chemical Society · 2023
    Review
  15. Review
  16. Article
  17. Article
  18. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

14 authors.

Kevin De AngeliOak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37830, USA; The Bredesen Center, The University of Tennessee, 821 Volunteer Blvd. Knoxville, TN 37996, USA. Electronic address: kevindeangeli94@gmail.com.
Shang GaoOak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37830, USA.
Ioana DanciuOak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37830, USA; Department of Biomedical Informatics, Vanderbilt University, 2525 West End Avenue, Nashville, TN 37203, USA.
Eric B DurbinCollege of Medicine, University of Kentucky, Lexington, KY 40536, USA.
Xiao-Cheng WuLouisiana State University Health Sciences Center, School of Public Health, New Orleans, LA 70112, USA.
Antoinette StroupRutgers Cancer Institute of New Jersey, New Brunswick, NJ, 08901, USA.
Jennifer DohertyHuntsman Cancer Institute, University of Utah, Salt Lake City, UT 84132, USA.
Stephen SchwartzFred Hutchinson Cancer Research Center, Epidemiology Program, Seattle, WA 98109, USA.
Charles WigginsUniversity of New Mexico, Albuquerque, NM 87131, USA.
Mark DamesynCalifornia Department of Public Health, Sacramento, CA 59814, USA.
Linda CoyleInformation Management Services Inc., Calverton, MD 20705, USA.
Lynne PenberthyNational Cancer Institute, Bethesda, MD 20814, USA.
Georgia D TourassiOak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37830, USA.
Hong-Jun YoonOak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN 37830, USA.

Funding

University of Kentucky Markey Cancer Center Support Grant ECIA SupplementP30CA177558 · NCI · UNIVERSITY OF KENTUCKY · PI KATHLEEN L. O'CONNOR · 2013 to 2026
$38.3M
POLICY & ENVIRONMENTAL APPROACHES & COMMUNITY-CLINICAL LINKAGES IN CANCER PROGRAMU58DP003907 · DP · UNIVERSITY OF KENTUCKY · PI TUCKER, THOMAS C · 2012 to 2014
$3.8M
NCCDPHP CDC HHS NU58DP006344NCCDPHP CDC HHS U58 DP003907NCI NIH HHS HHSN261201300021CNCI NIH HHS HHSN261201800007CNCI NIH HHS HHSN261201800009CNCI NIH HHS HHSN261201800009INCI NIH HHS HHSN261201800013CNCI NIH HHS HHSN261201800013INCI NIH HHS HHSN261201800014CNCI NIH HHS HHSN261201800014INCI NIH HHS HHSN261201800015CNCI NIH HHS HHSN261201800015INCI NIH HHS HHSN261201800016CNCI NIH HHS HHSN261201800016INCI NIH HHS HHSN261201800032CNCI NIH HHS HHSN261201800032INCI NIH HHS P30 CA177558NHLBI NIH HHS HHSN261201800013I
6 · The paper itself

Abstract

In the last decade, the widespread adoption of electronic health record documentation has created huge opportunities for information mining. Natural language processing (NLP) techniques using machine and deep learning are becoming increasingly widespread for information extraction tasks from unstructured clinical notes. Disparities in performance when deploying machine learning models in the real world have recently received considerable attention. In the clinical NLP domain, the robustness of convolutional neural networks (CNNs) for classifying cancer pathology reports under natural distribution shifts remains understudied. In this research, we aim to quantify and improve the performance of the CNN for text classification on out-of-distribution (OOD) datasets resulting from the natural evolution of clinical text in pathology reports. We identified class imbalance due to different prevalence of cancer types as one of the sources of performance drop and analyzed the impact of previous methods for addressing class imbalance when deploying models in real-world domains. Our results show that our novel class-specialized ensemble technique outperforms other methods for the classification of rare cancer types in terms of macro F1 scores. We also found that traditional ensemble methods perform better in top classes, leading to higher micro F1 scores. Based on our findings, we formulate a series of recommendations for other ML practitioners on how to build robust models with extremely imbalanced datasets in biomedical NLP applications.

Indexed as

Natural Language ProcessingNeoplasmsElectronic Health RecordsHumansMachine LearningNeural Networks, ComputerClass ImbalanceCNNDeep LearningEnsembleNLPText Classification

Identifiers

PMID34823030
PMCPMC9274264

What OpenQuestion holds

Textmetadata
LicenceTDM
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.