Deep learning-based multi-task prediction of response to neoadjuvant chemotherapy using multiscale whole slide images in breast cancer: A multicenter study.

Objective: Early predicting response before neoadjuvant chemotherapy (NAC) is crucial for personalized treatment plans for locally advanced breast cancer patients. We aim to develop a multi-task model using multiscale whole slide images (WSIs) features to predict the response to breast cancer NAC more finely. Methods: This work collected 1,670 whole slide images for training and validation sets, internal testing sets, external testing sets, and prospective testing sets of the weakly-supervised deep learning-based multi-task model (DLMM) in predicting treatment response and pCR to NAC. Our approach models two-by-two feature interactions across scales by employing concatenate fusion of single-scale feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. Results: In the retrospective analysis, DLMM exhibited excellent predictive performance for the prediction of treatment response, with area under the receiver operating characteristic curves (AUCs) of 0.869 [95% confidence interval (95% CI): 0.806-0.933] in the internal testing set and 0.841 (95% CI: 0.814-0.867) in the external testing sets. For the pCR prediction task, DLMM reached AUCs of 0.865 (95% CI: 0.763-0.964) in the internal testing and 0.821 (95% CI: 0.763-0.878) in the pooled external testing set. In the prospective testing study, DLMM also demonstrated favorable predictive performance, with AUCs of 0.829 (95% CI: 0.754-0.903) and 0.821 (95% CI: 0.692-0.949) in treatment response and pCR prediction, respectively. DLMM significantly outperformed the baseline models in all testing sets (P<0.05). Heatmaps were employed to interpret the decision-making basis of the model. Furthermore, it was discovered that high DLMM scores were associated with immune-related pathways and cells in the microenvironment during biological basis exploration. Conclusions: The DLMM represents a valuable tool that aids clinicians in selecting personalized treatment strategies for breast cancer patients.