Super-Identity Convolutional Neural Network for Face Hallucination

ECCV, pp. 196-211, 2018.

Cited by: 35|Bibtex|Views41|DOI:https://doi.org/10.1007/978-3-030-01252-6_12
EI
Other Links: dblp.uni-trier.de|academic.microsoft.com|arxiv.org
Weibo:
We present Super-Identity CNN to enhance the identity information during super resolving face images of size 12×14 pixels with an 8× upscaling factor

Abstract:

Face hallucination is a generative task to super-resolve the facial image with low resolution while human perception of face heavily relies on identity information. However, previous face hallucination approaches largely ignore facial identity recovery. This paper proposes Super-Identity Convolutional Neural Network (SICNN) to recover ide...More

Code:

Data:

0
Introduction
  • Face hallucination, which generates high-resolution (HR) facial images from lowresolution (LR) inputs, has attracted great interests in the past few years.
  • Fig. 1 shows some examples of hallucinated facial images generated by bicubic and several state-of-the-art methods.
  • Though they generate clearer facial images than bicubic, the identity similarities are still low, which means that they cannot recover accurate identity-related facial details.
  • Pixel-level cues cannot fully account for the perception process of the brain
  • These facts suggest that recovering identity information may improve both the recognizability and performance of hallucination
Highlights
  • Face hallucination, which generates high-resolution (HR) facial images from lowresolution (LR) inputs, has attracted great interests in the past few years
  • Motivated by the above observations, this paper proposes Super-Identity Convolutional Neural Network (SICNN) for identity-enhanced face hallucination
  • – We propose Super-identity Convolutional Neural Network (SICNN) for enhancing the identity information in face hallucination
  • – We propose Domain-Integrated Training method to overcome the problem caused by dynamic domain divergence when training Super-Identity Convolutional Neural Network
  • We report the results of average Peak Signalto-Noise Ratio (PSNR) and Structural Similarity (SSIM) in Tab. 3
  • We present Super-Identity CNN (SICNN) to enhance the identity information during super resolving face images of size 12×14 pixels with an 8× upscaling factor
Methods
  • Method Bicubic Ma et al LapSRN

    UR-DGN UR-DGN* SICNN SSIM. Bicubic Ma et al LapSRN UR-DGN UR-DGN* SICNN LFW Acc.
  • Human [28] [27] [26] [22] [31] [18]
Results
  • Evaluation on Face Hallucination

    The authors compare SICNN with other state-of-the-art methods and bicubic interpolation on Set C for face hallucination.
  • The authors observe that UR-DGN, trained by pixels-wise loss and adversarial loss, even shows inferior performance than LapSRN though with sharper visual results (See Sec. 4.4).
  • It means that UR-DGN will lose some identity information while super-resolving a face because the adversarial loss is not a pair-wise loss.
  • The authors tried using unaligned faces in training and testing and the proposed method still can achieve similar improvement of performance
Conclusion
  • The authors present Super-Identity CNN (SICNN) to enhance the identity information during super resolving face images of size 12×14 pixels with an 8× upscaling factor.
  • SICNN aims to minimize the identity difference between the hallucinated face and its corresponding HR face.
  • The authors propose a domain-integrated training approach to overcome the dynamic domain divergence problem when training SICNN.
  • Extensive experiments demonstrate that SICNN achieves superior hallucination results and significantly improves the performance of low-resolution face recognition
Summary
  • Introduction:

    Face hallucination, which generates high-resolution (HR) facial images from lowresolution (LR) inputs, has attracted great interests in the past few years.
  • Fig. 1 shows some examples of hallucinated facial images generated by bicubic and several state-of-the-art methods.
  • Though they generate clearer facial images than bicubic, the identity similarities are still low, which means that they cannot recover accurate identity-related facial details.
  • Pixel-level cues cannot fully account for the perception process of the brain
  • These facts suggest that recovering identity information may improve both the recognizability and performance of hallucination
  • Methods:

    Method Bicubic Ma et al LapSRN

    UR-DGN UR-DGN* SICNN SSIM. Bicubic Ma et al LapSRN UR-DGN UR-DGN* SICNN LFW Acc.
  • Human [28] [27] [26] [22] [31] [18]
  • Results:

    Evaluation on Face Hallucination

    The authors compare SICNN with other state-of-the-art methods and bicubic interpolation on Set C for face hallucination.
  • The authors observe that UR-DGN, trained by pixels-wise loss and adversarial loss, even shows inferior performance than LapSRN though with sharper visual results (See Sec. 4.4).
  • It means that UR-DGN will lose some identity information while super-resolving a face because the adversarial loss is not a pair-wise loss.
  • The authors tried using unaligned faces in training and testing and the proposed method still can achieve similar improvement of performance
  • Conclusion:

    The authors present Super-Identity CNN (SICNN) to enhance the identity information during super resolving face images of size 12×14 pixels with an 8× upscaling factor.
  • SICNN aims to minimize the identity difference between the hallucinated face and its corresponding HR face.
  • The authors propose a domain-integrated training approach to overcome the dynamic domain divergence problem when training SICNN.
  • Extensive experiments demonstrate that SICNN achieves superior hallucination results and significantly improves the performance of low-resolution face recognition
Tables
  • Table1: Quantitative comparison of different α on identity recovery and identity recognizability evaluation. Larger α brings better performance and it is stable when α is larger than 8
  • Table2: Quantitative comparison of different training approaches on identity recovery and identity recognizability evaluation. The results demonstrate the superiority of our proposed domain-integrated training
  • Table3: Quantitative hallucination comparison of different training approaches
  • Table4: Quantitative comparison on identity recovery and identity recognizability evaluation. The results demonstrate the superiority of our proposed method
  • Table5: Face verification performance of different methods on LFW [<a class="ref-link" id="c11" href="#r11">11</a>] and YTF [<a class="ref-link" id="c32" href="#r32">32</a>] benchmark. It shows that our method can help the recognition model to archive high accuracy with ultra-low-resolution inputs
Download tables as Excel
Related work
  • Single image super-resolution (SR) aims at recovering a HR image from a LR one. Face hallucination is a kind of class-specific image SR, which exploits the statistical properties of facial images. We classify face hallucination methods into two categories: classical approaches and deep learning approach.

    Classical Approaches. Subspace-based and facial components-based methods are two main kinds of classical face hallucination approaches.

    For subspace-based methods. Liu et al [17] employed a Principal Component Analysis (PCA) based global appearance model to hallucinate LR faces and a local non-parametric model to enhance the details. Ma et al [21] used multiple local exemplar patches sampled from aligned HR facial images to hallucinate LR faces. Li et al [16] resolved to sparse representation on local face patches. These subspace-based methods require precisely aligned reference HR and LR facial images with the same pose and facial expression.
Funding
  • This work was supported in part by MediaTek Inc and the Ministry of Science and Technology, Taiwan, under Grant MOST 107-2634-F-002 -007
  • We also benefit from the grants from NVIDIA and the NVIDIA DGX-1 AI Supercomputer
Reference
  • Bansal, A., Nanduri, A., Castillo, C., Ranjan, R., Chellappa, R.: Umdfaces: An annotated face dataset for training deep networks. arXiv:1611.01484 (2016)
    Findings
  • Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. ICLR (2016)
    Google ScholarLocate open access versionFindings
  • Chang, L., Tsao, D.Y.: The code for facial identity in the primate brain. Cell 169(6), 1013–1028 (2017)
    Google ScholarLocate open access versionFindings
  • Chen, B.C., Chen, C.S., Hsu, W.H.: Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. TMM 17(6), 804–815 (2015)
    Google ScholarLocate open access versionFindings
  • Chunjie, L., Qiang, Y., et al.: Cosine normalization: Using cosine similarity instead of dot product in neural networks. arXiv (2017)
    Google ScholarLocate open access versionFindings
  • Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: ECCV. pp. 391–407 (2016)
    Google ScholarFindings
  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS. pp. 2672–2680 (2014)
    Google ScholarFindings
  • He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In: ICCV. pp. 1026–1034 (2015)
    Google ScholarFindings
  • He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
    Google ScholarFindings
  • Huang, G., Liu, Z., Weinberge, r.K., Maaten, L.v.d.: Densely connected convolutional networks (2017)
    Google ScholarFindings
  • Huang, G.B., Learned-Miller, E.: Labeled faces in the wild: Updates and new reporting procedures. Dept. Comput. Sci., Univ. Massachusetts Amherst, Amherst, MA, USA, Tech. Rep pp. 14–003 (2014)
    Google ScholarLocate open access versionFindings
  • Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV. pp. 694–711 (2016)
    Google ScholarFindings
  • Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. pp. 1097–1105 (2012)
    Google ScholarFindings
  • Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution. CVPR (2017)
    Google ScholarLocate open access versionFindings
  • Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image superresolution using a generative adversarial network (2017)
    Google ScholarFindings
  • Li, Y., Cai, C., Qiu, G., Lam, K.M.: Face hallucination based on sparse local-pixel structure. PR 47(3), 1261–1270 (2014)
    Google ScholarLocate open access versionFindings
  • Liu, C., Shum, H.Y., Freeman, W.T.: Face hallucination: Theory and practice. IJCV 75(1), 115–134 (2007)
    Google ScholarLocate open access versionFindings
  • Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: Sphereface: Deep hypersphere embedding for face recognition (2017)
    Google ScholarFindings
  • Liu, W., Zhang, Y.M., Li, X., Yu, Z., Dai, B., Zhao, T., Song, L.: Deep hyperspherical learning. In: NIPS. pp. 3953–3963 (2017)
    Google ScholarFindings
  • Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV. pp. 3730–3738 (2015)
    Google ScholarLocate open access versionFindings
  • Ma, X., Zhang, J., Qi, C.: Hallucinating face by position-patch. PR 43(6), 2224– 2236 (2010)
    Google ScholarLocate open access versionFindings
  • Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: BMVC. vol. 1, p. 6 (2015)
    Google ScholarFindings
  • Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: NIPS. pp. 91–99 (2015)
    Google ScholarFindings
  • Rippel, O., Paluri, M., Dollar, P., Bourdev, L.: Metric learning with adaptive density discrimination. ICLR (2016)
    Google ScholarLocate open access versionFindings
  • Sajjadi, M.S., Scholkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: ICCV. pp. 4491–4500 (2017)
    Google ScholarFindings
  • Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. In: CVPR. pp. 815–823 (2015)
    Google ScholarFindings
  • Sun, Y., Wang, X., Tang, X.: Deeply learned face representations are sparse, selective, and robust. In: CVPR. pp. 2892–2900 (2015)
    Google ScholarFindings
  • Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: Closing the gap to humanlevel performance in face verification. In: CVPR. pp. 1701–1708 (2014)
    Google ScholarFindings
  • Tappen, M.F., Liu, C.: A bayesian approach to alignment-based image hallucination. In: ECCV. pp. 236–249 (2012)
    Google ScholarFindings
  • Van Der Maaten, L.: Accelerating t-sne using tree-based algorithms. JMLR 15(1), 3221–3245 (2014)
    Google ScholarLocate open access versionFindings
  • Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: ECCV. pp. 499–515 (2016)
    Google ScholarFindings
  • Wolf, L., Hassner, T., Maoz, I.: Face recognition in unconstrained videos with matched background similarity. In: CVPR. pp. 529–534 (2011)
    Google ScholarFindings
  • Yang, C.Y., Liu, S., Yang, M.H.: Structured face hallucination. In: ECVP. pp. 1099–1106 (2013)
    Google ScholarFindings
  • Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. arXiv:1411.7923 (2014)
    Findings
  • Yu, X., Porikli, F.: Ultra-resolving face images by discriminative generative networks. In: ECCV. pp. 318–333 (2016)
    Google ScholarFindings
  • Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. SPL 23(10), 1499–1503 (2016)
    Google ScholarLocate open access versionFindings
  • Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Learning face hallucination in the wild. In: AAAI. pp. 3871–3877 (2015)
    Google ScholarFindings
  • Zhu, S., Liu, S., Loy, C.C., Tang, X.: Deep cascaded bi-network for face hallucination. In: ECCV. pp. 614–630 (2016)
    Google ScholarFindings
Full Text
Your rating :
0

 

Tags
Comments