Laparoscopic augmented reality navigation system based on deep learning and SLAM.

The application of video see-through augmented reality (VST-AR) navigation in laparoscopic surgery helps visualize key anatomical structures that are hidden, which can enhance the surgeon’s intraoperative perception and improve the safety of the surgery. In this paper, a VST-AR navigation system oriented towards laparoscopic surgery is proposed. This system only relies on preoperative medical imaging slices, without the need for additional equipment. Novel and robust solutions are proposed for two main sub-problems: non-invasive registration and registration updating. Firstly, a vision Transformer-based depth estimation network is fine-tuned for 3D reconstruction of the intraoperative scene, which enhances its robustness to texture-less and variable soft tissues. Then, virtual markers are constructed to establish a spatial relationship between the virtual and real organs, achieving non-invasive registration. To solve the registration updating, based on the depth information obtained from network inference, RGBD SLAM (simultaneous localization and mapping) technology is introduced into the pose tracking of the stereoscopic laparoscope, realizing the synchronous updating of real and virtual laparoscopic viewpoints. The performance of the proposed navigation system on a phantom gallbladder is quantitatively evaluated by surgeons to determine its accuracy and potential for clinical application. The experimental results demonstrated reliable performance in depth estimation (average error 19.575 pixels), dynamic registration accuracy (2.23 ± 0.25 mm), and tracking precision (2.3 mm translation, [Formula: see text] rotation), indicating that the proposed navigation system can effectively guide the identification of key anatomical structures and improve the accuracy and success rate of laparoscopic surgeries.