Vision-based augmented reality guidance for setting up robot-assisted spine surgery.

purposeRobot-assisted procedures improve precision and patient outcomes, but setup often requires tedious manual positioning due to a lack of intuitive guidance. Augmented reality (AR) can assist with coarse alignment by providing visual cues in the correct anatomical context. However, most AR systems require additional setup steps to calibrate the head-mounted displays and imaging systems. We propose an AR guidance approach that uses only RGB video to perform spatial calibration with minimal setup effort.

methodsWe establish a shared coordinate frame between the AR headset and the intraoperative imaging system to enable markerless AR guidance. Inter-device poses are estimated directly from RGB video using a Vision Foundation Model (VFM). To improve calibration accuracy in challenging scenarios, such as partial view overlap and long baselines, we apply multi-frame Umeyama optimization with RANSAC, jointly refining scale, rotation, and translation. This process allows users to align the physical robot with the virtual surgical plan with the necessary accuracy.

resultsWe evaluate calibration and robot alignment accuracy against an ArUco-based baseline, achieving sub-3

conclusionExperimental results show that our vision-based calibration enables accurate needle alignment within the robot's workspace while eliminating setup complexity and fiducial dependence. This demonstrates the feasibility of seamless, markerless AR guidance for robotic procedures via vision-based calibration.