The RoDEM benchmark: evaluating the robustness of monocular single-shot depth estimation methods in minimally-invasive surgery.

purposeMonocular Single-shot Depth Estimation (MoSDE) methods for Minimally-Invasive Surgery (MIS) are promising but their robustness in surgical conditions remains questionable. We introduce the RoDEM benchmark, comprising an advanced analysis of perturbations, a dataset acquired in realistic MIS conditions and metrics. The dataset consists of 29,803 ex-vivo images including 44 video sequences with depth Ground-Truth covering clean conditions and nine perturbations. We give the performance evaluation of nine existing MoSDE methods.

methodsAn RGB-D structured-light camera was firmly attached to a laparoscope. The two cameras were internally calibrated and the rigid transformation between them was estimated. Synchronised images and videos were captured while producing real perturbations in three settings. The depth maps were eventually transferred to the laparoscope viewpoint and the images categorised by perturbation severity.

resultsThe proposed metrics cover accuracy (clean condition performance) and robustness (resilience to perturbations). We found that foundation models demonstrated higher accuracy than the other methods. All methods were robust to motion blur and bright light. Methods trained on large datasets were robust against smoke, blood, and low light whereas the other methods exhibited reduced robustness. None of the methods coped with lens dirtiness and defocus blur.

conclusionThis study highlighted the importance of robustness evaluation in MoSDE as many existing methods showed reduced accuracy against common surgical perturbations. It emphasises the importance of training with large datasets including perturbations. The proposed benchmark gives a precise and detailed analysis of a method's performance in the MIS conditions. It will be made publicly available.