Deep learning combining imaging, dose and clinical data for predicting bowel toxicity after pelvic radiotherapy.

Background and Purpose: A comprehensive understanding of radiotherapy toxicity requires analysis of multimodal data. However, it is challenging to develop a model that can analyse both 3D imaging and clinical data simultaneously. In this study, a deep learning model is proposed for simultaneously analysing computed tomography scans, dose distributions, and clinical metadata to predict toxicity, and identify the impact of clinical risk factors and anatomical regions. Materials and methods: : A deep model based on multiple instance learning with feature-level fusion and attention was developed. The study used a dataset of 313 patients treated with 3D conformal radiation therapy and volumetric modulated arc therapy, with heterogeneous cohorts varying in dose, volume, fractionation, concomitant therapies, and follow-up periods. The dataset included 3D computed tomography scans, planned dose distributions to the bowel cavity, and patient clinical data. The model was trained on patient-reported data on late bowel toxicity. Results: Results showed that the network can identify potential risk factors and critical anatomical regions. Analysis of clinical data jointly with imaging and dose for bowel urgency and faecal incontinence improved performance (area under receiver operating characteristic curve [AUC] of 88% and 78%, respectively) while best performance for diarrhoea was when analysing clinical features alone (68% AUC). Conclusions: Results demonstrated that feature-level fusion along with attention enables the network to analyse multimodal data. This method also provides explanations for each input's contribution to the final result and detects spatial associations of toxicity.