D2polyp-Net: A cross-modal space-guided network for real-time colorectal polyp detection and diagnosis

Colorectal polyp screening and identification during endoscopy are crucial for reducing cancer morbidity and patient mortality. Although some deep neural networks for polyp detection and diagnosis have been proposed, there are still problems in small-size polyp localization, poor classification performance especially in white-light endoscopic imaging modality, as well as real -time performance. In this paper, we propose D2polyp-Net, employing a space-guided localization framework and a cross-modal pairwise training strategy, which performs real -time polyp detection and diagnosis simultaneously. The space-guided framework consisting of a dualpyramid structure builds a fusion bridge between shallow-spatial and deep-semantic information, achieving more accurate localization and providing effective spatial information for polyp classification. The cross-modal pairwise training strategy constrains the output feature of white-light images to approach that of the corresponding narrow-band images, as the latter provide richer mucosal structure and blood flow information to classify polyps. The above modules are validated on publicly available datasets, and D2polyp-Net is demonstrated to surpass other compared state-of-the-art algorithms. It achieves an mAP@0.5 of 0.817 at a frame rate of 28 FPS, with precision and recall above 0.7 for both hyperplastic and adenomatous polyps. Cross-center tests on clinical data further demonstrate the generalization ability of D2polyp-Net, with the mAP@0.5 reaching 0.780. The experimental results show that D2polyp-Net can assist physicians in polyp detection and diagnosis during clinical colonoscopy, which is important for reducing the polyp miss rate and improving the diagnostic efficiency.