Identify characteristics of Vietnamese oral squamous cell carcinoma patients by machine learning on transcriptome and clinical-histopathological analysis.

Background/purpose: Oral squamous cell carcinoma (OSCC) is notorious for its low survival rates, due to the advanced stage at which it is commonly diagnosed. To enhance early detection and improve prognostic assessments, our study harnesses the power of machine learning (ML) to dissect and interpret complex patterns within mRNA-sequencing (RNA-seq) data and clinical-histopathological features. Materials and methods: 206 retrospective Vietnamese OSCC formalin-fixed paraffin-embedded (FFPE) tumor samples, of which 101 were subjected to RNA-seq for classification based on gene expression. Then, learning models were built based on clinical-histopathological data to predict OSCC subtypes and propose potential biomarkers for the remaining 105 samples. Results: 2 distinct groups of OSCC with different clinical-histopathological characteristics and gene expression. Subgroup 1 was characterized by severe histopathologic features with immune response and apoptosis signatures while subgroup 2 was denoted by more clinical/pathological features, cell division and malignant signatures. XGBoost and SVM (Support Vector Machine) models showed the best performance in predicting subtype OSCC. The study also proposed 12 candidate genes as potential biomarkers for OSCC subtypes (6/group). Conclusion: The study identified characteristics of Vietnamese OSCC patients through a combination of mRNA sequencing and clinical-histopathological analysis. It contributes to the insight into the tumor microenvironment of OSCC and provides accurate ML models for biomarker prediction using clinical-histopathological features.