Two-Channel VAE-GAN Based Image-To-Video Translation

We propose a VAE-GAN network with a two-channel decoder for addressing multiple image-to-video translation tasks, i.e., generating multiple videos of different categories by a single model. We consider this image-to-video translation as a video generation task rather than a video prediction that needs multiple frames as input. After training, the model only requires the first frame of the video and its corresponding attribute to generate the required video. The advantage of combining the Variational Autoencoder (VAE) and Generative Adversarial Network (GAN) is to avoid the shortcomings of both: VAE components can give rise to blur, and unstable gradients caused by the GAN. Extensive qualitative and quantitative experiments are conducted on the MUG [1] dataset. We draw the following conclusions from this empirical study: compared with state-of-the-art approaches, our approach (VAE-GAN) exhibits significant improvements in generative capability.