Camera augmentation: enabling uncalibrated stereo matching of minimally invasive surgery images by training from the wealth of public synthetic image datasets.

purposeExisting models for stereo matching in minimally invasive surgery (MIS) require calibrated stereo images. Accurate calibration is however often unavailable intraoperatively. Training an uncalibrated stereo model is thus attractive but challenging owing to the lack of disparity-labelled surgical images.

methodsWe leverage the wealth of non-medical stereo synthetic image datasets. These data were however generated in ideal conditions-rectified and with centred principal points-hence differ from real uncalibrated MIS images. We propose camera augmentation, a new type of image augmentation that augments a dataset by altering the camera's orientation and intrinsic parameters via geometric parameters. We augment the idealised existing datasets, sampling the geometric augmentation parameters from distributions estimated through an in-depth analysis and modelling of stereo laparoscopes. This forms the camera augmentation training strategy (CATS), with which we retrain RAFTStereo and IGEV++ for zero-shot uncalibrated stereo matching in MIS.

resultsWe evaluated using the SCARED, StereoMIS, RIS2017, and an in-house datasets. In the uncalibrated setting on the SCARED dataset, CATS-RAFTStereo and CATS-IGEV++ achieved end-point errors (EPE) of 1.42 and 1.41 pixels. This is a successful result, as the reference pretrained models obtained 1.23 and 1.21 pixels in the calibrated setting and failed in the uncalibrated setting.

conclusionCamera augmentation bridges the gap between ideally conditioned datasets and the real surgical conditions of uncertain or unavailable calibration, enabling the retraining of state-of-the-art architectures. Beyond stereo, the proposed CATS is applicable to tasks sensitive to camera geometry. Code and models will be released publicly.