We introduce a novel approach for high-resolution talking head generation from a single image and audio input. Prior methods using explicit face models, like 3D morphable models (3DMM) and facial landmarks, often fall short in generating high-fidelity videos due to their lack of appearance-aware motion representation. While generative approaches such as video diffusion models achieve high video quality, their slow processing speeds limit practical application. Our proposed model, Implicit Face Motion Diffusion Model (IF-MDM), employs implicit motion to encode human faces into appearance-aware compressed facial latents, enhancing video generation. Although implicit motion lacks the spatial disentanglement of explicit models, which complicates alignment with subtle lip movements, we introduce motion statistics to help capture fine-grained motion information. Additionally, our model provides motion controllability to optimize the trade-off between motion intensity and visual quality during inference. IF-MDM supports real-time generation of 512x512 resolution videos at up to 45 frames per second (fps). Extensive evaluations demonstrate its superior performance over existing diffusion and explicit face models. The code will be released publicly, available alongside supplementary materials. The video results can be found on https://bit.ly/ifmdm_supplementary.

Dense Interspecies Face Embedding (DIFE) is a new direction for understanding faces of various animals by extracting common features among animal faces including human face. There are three main obstacles for interspecies face understanding: (1) lack of animal data compared to human, (2) ambiguous connection between faces of various animals, and (3) extreme shape and style variance. To cope with the lack of data, we utilize multi-teacher knowledge distillation of CSE and StyleGAN2 requiring no additional data or label. Then we synthesize pseudo pair images through the latent space exploration of StyleGAN2 to find implicit associations between different animal faces. Finally, we introduce the semantic matching loss to overcome the problem of extreme shape differences between species. To quantitatively evaluate our method over possible previous methodologies like unsupervised keypoint detection, we perform interspecies facial keypoint transfer on MAFL and AP-10K. Furthermore, the results of other applications like interspecies face image manipulation and dense keypoint transfer are provided. The code is available at https://github.com/kingsj0405/dife.

This paper reviews the NTIRE 2021 challenge on learning the super-Resolution space. It focuses on the participating methods and final results. The challenge addresses the problem of learning a model capable of predicting the space of plausible super-resolution (SR) images, from a single low-resolution image. The model must thus be capable of sampling diverse outputs, rather than just generating a single SR image. The goal of the challenge is to spur research into developing learning formulations and models better suited for the highly ill-posed SR problem. And thereby advance the state-of-the-art in the broader SR field. In order to evaluate the quality of the predicted SR space, we propose a new evaluation metric and perform a comprehensive analysis of the participating methods. The challenge contains two tracks: 4× and 8 scale factor. In total, 11 teams competed in the final testing× phase.

Multiple high-resolution (HR) images can be generated from a single low-resolution (LR) image, as super-resolution (SR) is an underdetermined problem. Recently, the conditional normalizing flow-based model, SRFlow, shows remarkable performance by learning an exact map-ping from HR image manifold to a latent space. The flow-based SR model allows sampling multiple output images from a learned SR space with a given LR image. In this work, we propose SRFlow-DA which has a more suitable architecture for the SR task based on the original SRFlow model. Specifically, our approach enlarges the receptive field by stacking more convolutional layers in the affine couplings, and so our model can get more expressive power. At the same time, we reduce the total number of model pa-rameters for efficiency. Compared to SRFlow, our SRFlow-DA achieves better or comparable PSNR and LPIPS for × 4 and ×8 SR tasks, while having a reduced number of parameters. In addition, our method generates visually clear results without excessive sharpness artifacts.

The performance of image super-resolution (SR) has been greatly improved by using convolutional neural networks. Most of the previous SR methods have been studied up to ×4 upsampling, and few were studied for ×16 upsampling. The general approach for perceptual ×4 SR is using GAN with VGG based perceptual loss, however, we found that it creates inconsistent details for perceptual ×16 SR. To this end, we have investigated loss functions and we propose to use GAN with LPIPS [23] loss for perceptual extreme SR. In addition, we use U-net structure discriminator [14] together to consider both the global and local context of an input image. Experimental results show that our method outperforms the conventional perceptual loss, and we achieved second and first place in the LPIPS and PI measures respectively for NTIRE 2020 perceptual extreme SR challenge.

This paper reviews the NTIRE 2020 challenge on perceptual extreme super-resolution with focus on proposed solutions and results. The challenge task was to super-resolve an input image with a magnification factor ×16 based on a set of prior examples of low and corresponding high resolution images. The goal is to obtain a network design capable to produce high resolution results with the best perceptual quality and similar to the ground truth. The track had 280 registered participants, and19 teams submitted the final results. They gauge the state-of-the-art in single image super-resolution.