Image Captioning Via Masked Conditional Diffusion.

Current image captioning methods mainly adopt the autoregressive framework that operates through a next-token prediction paradigm. A non-autoregressive method called diffusion models has shown superiority in image generation. However, their potential in image captioning remains underexplored due to the visual-language misalignment. In this work, we present a novel Masked Conditional Diffusion model (MC-Diffusion). It contains a discrete denoising diffusion probabilistic model (D3PM) and a pre-trained vector quantized variational autoencoder (VQ-VAE). Specifically, we first extract discrete image features via VQ-VAE. Conditioned on these discrete image features, the discrete diffusion model generates captions through transformer blocks to establish discrete-to-discrete alignment. Furthermore, we propose a simple yet effective guidance method, named Masked Condition Strategy (MCS). Compared with classifier-free guidance, our proposed method achieves finer-grained visual-language alignment while demonstrating superior capability in model guidance. Experiments on the CUB-200 dataset show that the proposed method performs better than baselines on several metrics. Compared with classifier-free guidance, MCS achieves similar performance on reference-based metrics (e.g., BLEU, Meteor, etc.) while alleviating the hurt on CLIPScore.