Multi-scale Multimodal Representation for Enhanced Survival Prediction in Computational Pathology.

Prognostic prediction in computational pathology integrates diverse multimodal data, including pathological images and transcriptomic profiles, to accurately forecast patient outcomes. Although the potential for multimodal integration to enhance predictive accuracy, efficiently representing high-dimensional whole slide images (WSIs) and transcriptomic data remains challenging. Here, we propose the multi-scale multimodal representation (MSMR) framework, designed to integrate multi-scale pathological features from WSIs with transcriptomic data encoded via biological pathways. MSMR employs hierarchical clustering to effectively extract morphological features from WSIs at multiple magnifications and utilizes a biologically-informed encoder to map transcriptomic data onto relevant pathways, thereby capturing critical gene functions and interactions. These multimodal features are then fused into compact tokens and modeled with a Bernstein polynomial-based Kolmogorov-Arnold (BKA) regressor, enabling the detection of intricate spatial-genetic interactions associated with patient survival. Comprehensive evaluation on six datasets from The Cancer Genome Atlas (TCGA) demonstrates that MSMR significantly outperforms existing methods, establishing new benchmarks for survival prediction accuracy. Furthermore, MSMR’s interpretability through visualization and statistical analyses reveals meaningful histogenomic interactions, offering a promising computational tool for personalized treatment planning and informed clinical decision-making. Source code is available at https://github.com/kennysyp/MSMR .