Deep Regression with Spatial-Frequency Feature Coupling and Image Synthesis for Robot-Assisted Endomicroscopy

Probe-based confocal laser endomicroscopy (pCLE) allows in-situ visualisation of cellular morphology for intraoperative tissue characterization. Robotic manipulation of the pCLE probe can maintain the probe-tissue contact within micrometre working range to achieve the precision and stability required to capture good quality microscopic information. In this paper, we propose the first approach to automatically regress the distance between a pCLE probe and the tissue surface during robotic tissue scanning. The Spatial-Frequency Feature Coupling network (SFFC-Net) was designed to regress probe-tissue distance by extracting an enhanced data representation based on the fusion of spatial and frequency domain features. Image-level supervision is used in a novel fashion in regression to enable the network to effectively learn the relationship between the sharpness of the pCLE image and its distance from the tissue surface. Consequently, a novel Feedback Training (FT) module has been designed to synthesise unseen images to incorporate feedback into the training process. The first pCLE regression dataset (PRD) was generated which includes ex-vivo images with corresponding probe-tissue distance. Our performance evaluation verifies that the proposed network outperforms other state-of-the-art (SOTA) regression networks.