MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer

The Diffusion Probabilistic Model (DPM) has recently gained popularity in the field of computer vision, thanks to its image generation applications, such as Imagen, Latent Diffusion Models, and Stable Diffusion, which have demonstrated impressive capabilities and sparked much discussion within the community. Recent studies have also found DPM to be useful in the field of medical image analysis, as evidenced by the strong performance of the medical image segmentation model MedSegDiff in various tasks. While these models were originally designed with a UNet backbone, they may also potentially benefit from the incorporation of vision transformer techniques. However, we discovered that simply combining these two approaches resulted in subpar performance. In this paper, we propose a novel transformer-based conditional UNet framework, as well as a new Spectrum-Space Transformer (SS-Former) to model the interaction between noise and semantic features. This architectural improvement leads to a new diffusion-based medical image segmentation method called MedSegDiff-V2, which significantly improves the performance of MedSegDiff. We have verified the effectiveness of MedSegDiff-V2 on eighteen organs of five segmentation datasets with different image modalities. Our experimental results demonstrate that MedSegDiff-V2 outperforms state-of-the-art (SOTA) methods by a considerable margin, further proving the generalizability and effectiveness of the proposed model.