Perseus: perception with semantic endoscopic understanding and SLAM.

purposeNatural orifice surgeries minimize the need for incisions and reduce recovery time compared to open surgery; however, they require higher expertise due to visualization and orientation challenges. To enable reliable scene understanding for surgeon guidance and automation, we propose a perception pipeline that generates semantically informed 3D reconstructions.

methodsWe bring learning-based segmentation, depth estimation, and 3D reconstruction modules together. Through segmentation, we delineate tumor and prostate lobe borders, and through depth estimation and real-time SLAM-based reconstruction, we generate dense 3D point clouds from monocular videos. By propagating 2D labels into 3D space, we create real-time segmented maps of the surgical scenes. Additionally, we use registration with robot poses to solve the scale ambiguity of mapping from monocular images and allow the use of semantically informed real-time reconstructions in robotic surgeries.

resultsWe achieve sub-millimeter reconstruction accuracy based on average one-sided Chamfer distances, average pose registration RMSE of 0.9 mm, and an estimated scale within 2% of ground truth. Compared to offline Structure-from-Motion baseline, the proposed SLAM-based approach improves processing time while maintaining or improving reconstruction accuracy. Qualitative evaluations show robustness in challenging scenarios, including submerged prostate experiments and cadaver airway explorations.

conclusionWe present a modular perception pipeline, integrating semantic segmentation with real-time monocular SLAM for natural orifice surgeries. This pipeline offers a promising solution for scene understanding that can facilitate automation or surgeon guidance. Due to its plug-and-play design and demonstrated generalizability across anatomies and experimental conditions, this framework provides a scalable foundation for future clinical translation.