Hybrid CNN-transformer demosaicing for bioinspired single-chip color-near-infrared fluorescence imaging in oncologic surgery.

Significance: Single-chip multispectral imaging sensors with vertically stacked photodiodes and pixelated spectral filters enable advanced, real-time visualization for image-guided cancer surgery. However, their design inherently reduces spatial resolution. We present a convolutional neural network (CNN)-transformer demosaicing algorithm, validated on both clinical and preclinical datasets that effectively doubles spatial resolution and improves image quality-substantially enhancing intraoperative cancer visualization. Aim: We present a CNN-transformer-based demosaicing approach specifically optimized for reconstructing high-resolution color and NIR images acquired by a hexachromatic imaging sensor. Approach: A hybrid CNN-transformer demosaicing model was developed and trained on color-image datasets, then rigorously evaluated on color and NIR images to demonstrate superior reconstruction quality compared with conventional bilinear interpolation and residual CNN methods. Results: Our CNN-transformer demosaicing method achieves an average mean squared error (MSE) reduction of Conclusions: We demonstrate improvements in spatial resolution and image fidelity for color and NIR images obtained from hexachromatic imaging sensors, achieved by integrating convolutional neural networks with transformer architectures. Given recent advances in GPU computing, our CNN-transformer approach offers a practical, real-time solution for enhanced multispectral imaging during cancer surgery.