Feasibility of semantic segmentation of anatomical structures during minimally invasive lobectomy using a transformer-based deep learning architecture.

Objectives: During anatomic lung resection, intraoperative guidance identifying anatomic structures at risk of injury may improve surgical safety and serve as an educational tool. To enhance clinical practices, we aim to develop and evaluate the accuracy of semantic segmentation using a deep learning model capable of recognizing such key anatomic structures during right upper lobectomies. Methods: We collected recordings of 95 robotic-assisted right upper lobectomy cases for lung malignancy. The videos were divided into 2 sections: (1) subcarinal lymph node dissection (section 1) and (2) the subsequent phase (section 2). We labeled the following: bronchus, instruments, lung parenchyma, lymph node, pulmonary artery/vein, and vagus/phrenic nerve. Using these data, we trained and tested deep learning architectures for semantic segmentation and evaluated their performances using a Dice Similarity Coefficient score. Results: A total of 2901 annotated frames were divided among a training set (2355 frames), a hold-out testing set (546 frames), and a final testing set from an external dataset (600 frames). By using SegFormer as an encoder in a U-Net architecture, instruments and lung parenchyma were segmented with a Dice of 0.90 and 0.91 in section 1 and 0.91 and 0.93 in section 2, respectively. The model could not achieve high accuracy for bronchus and lymph node (Dice: ∼0.50), and vagus nerve (Dice: 0.25) in section 1. In section 2, the model has reasonably high segmentation accuracy for the bronchus (Dice: 0.57), pulmonary artery/vein (Dice: 0.78), and phrenic nerve (Dice: 0.66). Conclusions: We demonstrated the feasibility of semantic segmentation during right upper lobectomies using deep learning.