Learning a Single Convolutional Super-Resolution Network for Multiple Degradations

computer vision and pattern recognition, 2018.

Cited by: 224|Bibtex|Views133|DOI:https://doi.org/10.1109/cvpr.2018.00344
EI
Other Links: dblp.uni-trier.de|academic.microsoft.com|arxiv.org
Weibo:
Degradation maps are obtained by a simple dimensionality stretching of the degradation parameters

Abstract:

Recent years have witnessed the unprecedented success of deep convolutional neural networks (CNNs) in single image super-resolution (SISR). However, existing CNN-based SISR methods mostly assume that a low-resolution (LR) image is bicubicly downsampled from a high-resolution (HR) image, thus inevitably giving rise to poor performance when...More

Code:

Data:

0
Introduction
  • Single image super-resolution (SISR) aims to recover a high-resolution (HR) version of a low-resolution (LR) input.
  • SISR methods can be broadly classified into three categories, i.e., interpolation-based methods, model-based optimization methods and discriminative learning methods.
  • The integration of convolutional neural network (CNN) denoiser prior and model-based optimization can improve the efficiency to some extent, it still suffers from the typical drawbacks of model-based optimization methods, e.g., it is not in an end-to-end learning manner and involves hand-designed parameters [57].
  • Discriminative learning methods have attracted considerable attentions due to their favorable SISR performance in terms of effectiveness and efficiency.
  • Recent years have witnessed a dramatic upsurge of using CNN for SISR
Highlights
  • Single image super-resolution (SISR) aims to recover a high-resolution (HR) version of a low-resolution (LR) input
  • Where x⊗k represents the convolution between a blur kernel k and a latent HR image x, ↓s is a subsequent downsampling operation with scale factor s, and n usually is additive white Gaussian noise (AWGN) with standard deviation σ
  • The integration of convolutional neural network (CNN) denoiser prior and model-based optimization can improve the efficiency to some extent, it still suffers from the typical drawbacks of model-based optimization methods, e.g., it is not in an end-to-end learning manner and involves hand-designed parameters [57]
  • We proposed an effective super-resolution network with high scalability of handling multiple degradations via a single model
  • Degradation maps are obtained by a simple dimensionality stretching of the degradation parameters
  • The results on real LR images showed that the proposed method can reconstruct visually plausible HR images
Methods
  • Before solving the problem of SISR, it is important to have a clear understanding of the degradation model which is not limited to Eqn (1).
  • Another practical degradation model can be given by y = (x ↓s) ⊗ k + n.
  • When ↓ is the bicubic downsampler, Eqn (2) corresponds to a deblurring problem followed by a SISR problem with bicubic degradation.
  • The authors make a short discussion on blur kernel k, noise n and downsampler ↓
Conclusion
  • The authors proposed an effective super-resolution network with high scalability of handling multiple degradations via a single model.
  • Different from existing CNNbased SISR methods, the proposed super-resolver takes both LR image and its degradation maps as input.
  • The results on synthetic LR images demonstrated that the proposed super-resolver can produce state-of-the-art results on bicubic degradation and perform favorably on other degradations and even spatially variant degradations.
  • The proposed super-resolver offers a feasible solution toward practical CNN-based SISR applications
Summary
  • Introduction:

    Single image super-resolution (SISR) aims to recover a high-resolution (HR) version of a low-resolution (LR) input.
  • SISR methods can be broadly classified into three categories, i.e., interpolation-based methods, model-based optimization methods and discriminative learning methods.
  • The integration of convolutional neural network (CNN) denoiser prior and model-based optimization can improve the efficiency to some extent, it still suffers from the typical drawbacks of model-based optimization methods, e.g., it is not in an end-to-end learning manner and involves hand-designed parameters [57].
  • Discriminative learning methods have attracted considerable attentions due to their favorable SISR performance in terms of effectiveness and efficiency.
  • Recent years have witnessed a dramatic upsurge of using CNN for SISR
  • Objectives:

    It is natural to raise the following questions, which are the focus of the paper: (i) Can the authors learn a single model to effectively handle multiple and even spatially variant degradations? (ii) Is it possible to use synthetic data to train a model with high practicability? This work aims to make one of the first attempts towards answering these two questions.
  • It is natural to raise the following questions, which are the focus of the paper: (i) Can the authors learn a single model to effectively handle multiple and even spatially variant degradations?
  • (ii) Is it possible to use synthetic data to train a model with high practicability?
  • This work aims to make one of the first attempts towards answering these two questions.
  • Instead of handling the bicubic degradation only, the aim is to learn a single network to handle multiple degradations
  • Methods:

    Before solving the problem of SISR, it is important to have a clear understanding of the degradation model which is not limited to Eqn (1).
  • Another practical degradation model can be given by y = (x ↓s) ⊗ k + n.
  • When ↓ is the bicubic downsampler, Eqn (2) corresponds to a deblurring problem followed by a SISR problem with bicubic degradation.
  • The authors make a short discussion on blur kernel k, noise n and downsampler ↓
  • Conclusion:

    The authors proposed an effective super-resolution network with high scalability of handling multiple degradations via a single model.
  • Different from existing CNNbased SISR methods, the proposed super-resolver takes both LR image and its degradation maps as input.
  • The results on synthetic LR images demonstrated that the proposed super-resolver can produce state-of-the-art results on bicubic degradation and perform favorably on other degradations and even spatially variant degradations.
  • The proposed super-resolver offers a feasible solution toward practical CNN-based SISR applications
Tables
  • Table1: Average PSNR and SSIM results for bicubic degradation on datasets Set5 [<a class="ref-link" id="c3" href="#r3">3</a>], Set14 [<a class="ref-link" id="c54" href="#r54">54</a>], BSD100 [<a class="ref-link" id="c33" href="#r33">33</a>] and Urban100 [<a class="ref-link" id="c19" href="#r19">19</a>]. The best two results are highlighted in red and blue colors, respectively
  • Table2: Average PSNR and SSIM results of different methods with different degradations on Set5. The best results are highlighted in red color. The results highlighted in gray color indicate unfair comparison due to mismatched degradation assumption
Download tables as Excel
Related work
  • The first work of using CNN to solve SISR can be traced back to [8] where a three-layer super-resolution network (SRCNN) was proposed. In the extended work [9], the authors investigated the impact of depth on super-resolution and empirically showed that the difficulty of training deeper model hinders the performance improvement of CNN super-resolvers. To overcome the training difficulty, Kim et al [24] proposed a very deep super-resolution (VDSR) method with residual learning strategy. Interestingly, they showed that VDSR can handle multiple scales superresolution. By analyzing the relation between CNN and MAP inference, Zhang et al [56] pointed out that CNN mainly model the prior information and they empirically demonstrated that a single model can handle multiple scales super-resolution, image deblocking and image denoising. While achieving good performance, the above methods take the bicubicly interpolated LR image as input, which not only suffers from high computational cost but also hinders the effective expansion of receptive field.
Funding
  • This work is supported by National Natural Science Foundation of China (grant no. 61671182, 61471146), HK RGC General Research Fund (PolyU 152240/15E) and PolyU-Alibaba Collaborative Research Project “Quality Enhancement of Surveillance Images and Videos”
  • We gratefully acknowledge the support from NVIDIA Corporation for providing us the Titan Xp GPU used in this research
Study subjects and analysis
widely-used datasets: 4
However, in order to show the advantage of the dimensionality stretching strategy, the proposed method is also compared with other CNN-based methods specifically designed for bicubic degradation. Table 1 shows the PSNR and SSIM [50] results of stateof-the-art CNN-based SISR methods on four widely-used datasets. As one can see, SRMD achieves comparable results with VDSR at small scale factor and outperforms VDSR at large scale factor

Reference
  • E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, volume 3, pages 126–135, July 2017. 6
    Google ScholarLocate open access versionFindings
  • S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167–1183, 2001
    Google ScholarLocate open access versionFindings
  • M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. A. Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference, 2012. 7
    Google ScholarLocate open access versionFindings
  • S. A. Bigdeli, M. Jin, P. Favaro, and M. Zwicker. Deep meanshift priors for image restoration. In Advances in Neural Information Processing Systems, 2017. 1, 2
    Google ScholarLocate open access versionFindings
  • G. Boracchi and A. Foi. Modeling the performance of image restoration from motion blur. IEEE Transactions on Image Processing, 21(8):3502–3517, Aug 2012. 3
    Google ScholarLocate open access versionFindings
  • Y. Chen, W. Yu, and T. Pock. On learning optimized reaction diffusion processes for effective image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5261–5269, 2015. 2
    Google ScholarLocate open access versionFindings
  • Z. Cui, H. Chang, S. Shan, B. Zhong, and X. Chen. Deep network cascade for image super-resolution. In European Conference on Computer Vision, pages 49–64, 2014. 3
    Google ScholarLocate open access versionFindings
  • C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184–199, 2014. 2
    Google ScholarLocate open access versionFindings
  • C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2016. 2, 7
    Google ScholarLocate open access versionFindings
  • C. Dong, C. C. Loy, and X. Tang. Accelerating the superresolution convolutional neural network. In European Conference on Computer Vision, pages 391–407, 2016. 2
    Google ScholarLocate open access versionFindings
  • W. Dong, L. Zhang, G. Shi, and X. Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 22(4):1620–1630, 2013. 1, 2, 3, 6, 7
    Google ScholarLocate open access versionFindings
  • N. Efrat, D. Glasner, A. Apartsin, B. Nadler, and A. Levin. Accurate blur models vs. image priors in single image superresolution. In IEEE International Conference on Computer Vision, pages 2832–2839, 2013. 1, 3
    Google ScholarLocate open access versionFindings
  • K. Egiazarian and V. Katkovnik. Single image superresolution via BM3D sparse coding. In European Signal Processing Conference, pages 2849–2853, 2015. 1
    Google ScholarLocate open access versionFindings
  • W. Freeman and C. Liu. Markov random fields for superresolution and texture synthesis. Advances in Markov Random Fields for Vision and Image Processing, 1:155–165, 2011. 3
    Google ScholarLocate open access versionFindings
  • D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In IEEE International Conference on Computer Vision, pages 349–356, 2009. 3
    Google ScholarLocate open access versionFindings
  • I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014. 1, 2
    Google ScholarLocate open access versionFindings
  • H. He and W.-C. Siu. Single image super-resolution using Gaussian process regression. In IEEE Conference on Computer Vision and Pattern Recognition, pages 449–456, 2011. 3
    Google ScholarLocate open access versionFindings
  • K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 1
    Google ScholarLocate open access versionFindings
  • J.-B. Huang, A. Singh, and N. Ahuja. Single image superresolution from transformed self-exemplars. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2015. 7, 8
    Google ScholarLocate open access versionFindings
  • S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448– 456, 2015. 4, 5
    Google ScholarLocate open access versionFindings
  • M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Advances in Neural Information Processing Systems, pages 2017–2025, 2015. 1, 5
    Google ScholarLocate open access versionFindings
  • J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694–711, 2016. 2
    Google ScholarLocate open access versionFindings
  • J. Kim, J. Kwon Lee, and K. Mu Lee. Deeply-recursive convolutional network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1637–1645, 2016. 2
    Google ScholarLocate open access versionFindings
  • J. Kim, J. K. Lee, and K. M. Lee. Accurate image superresolution using very deep convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1646–1654, 2016. 2, 4, 7, 8
    Google ScholarLocate open access versionFindings
  • D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015. 5, 6
    Google ScholarLocate open access versionFindings
  • A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012. 4, 5, 6
    Google ScholarLocate open access versionFindings
  • W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate superresolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 624–632, July 2017. 2, 7
    Google ScholarLocate open access versionFindings
  • Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, 2015. 1
    Google ScholarLocate open access versionFindings
  • C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4681–4690, July 2017. 2, 5, 7
    Google ScholarLocate open access versionFindings
  • B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 136–144, July 2017. 2
    Google ScholarLocate open access versionFindings
  • K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing, 26(2):1004–1016, 2017. 6
    Google ScholarLocate open access versionFindings
  • J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2272–2279, 2009. 1
    Google ScholarLocate open access versionFindings
  • D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference on Computer Vision, volume 2, pages 416–423, July 2001. 6, 7
    Google ScholarLocate open access versionFindings
  • T. Meinhardt, M. Moller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In IEEE International Conference on Computer Vision, pages 1781–1790, 2017. 2
    Google ScholarLocate open access versionFindings
  • J. Pan, Z. Hu, Z. Su, and M.-H. Yang. Deblurring text images via L0-regularized intensity and gradient prior. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2908, 2014. 8
    Google ScholarLocate open access versionFindings
  • T. Peleg and M. Elad. A statistical prediction model based on sparse representations for single image super-resolution. IEEE transactions on Image Processing, 23(6):2569–2582, 2014. 3
    Google ScholarLocate open access versionFindings
  • J. S. Ren, L. Xu, Q. Yan, and W. Sun. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems, pages 901–909, 2015. 2
    Google ScholarLocate open access versionFindings
  • G. Riegler, S. Schulter, M. Ruther, and H. Bischof. Conditioned regression models for non-blind single image superresolution. In IEEE International Conference on Computer Vision, pages 522–530, 2015. 3, 5
    Google ScholarLocate open access versionFindings
  • Y. Romano, M. Elad, and P. Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4):1804–1844, 2017. 2
    Google ScholarLocate open access versionFindings
  • Y. Romano, J. Isidoro, and P. Milanfar. RAISR: rapid and accurate image super resolution. IEEE Transactions on Computational Imaging, 3(1):110–125, 2017. 1
    Google ScholarLocate open access versionFindings
  • W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1874–1883, 2016. 2, 4
    Google ScholarLocate open access versionFindings
  • Y. Shi, K. Wang, C. Chen, L. Xu, and L. Lin. Structurepreserving image super-resolution via contextualized multitask learning. IEEE Transactions on Multimedia, 2017. 2
    Google ScholarLocate open access versionFindings
  • A. Singh, F. Porikli, and N. Ahuja. Super-resolving noisy images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2846–2853, 2014. 3
    Google ScholarLocate open access versionFindings
  • Y. Tai, J. Yang, and X. Liu. Image super-resolution via deep recursive residual network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3147–3155, 2017. 2, 7
    Google ScholarLocate open access versionFindings
  • Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In IEEE International Conference on Computer Vision, pages 4539–4547, 2017. 2
    Google ScholarLocate open access versionFindings
  • R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang. Ntire 2017 challenge on single image superresolution: Methods and results. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 114–125, July 2017. 2
    Google ScholarLocate open access versionFindings
  • R. Timofte, V. De Smet, and L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian Conference on Computer Vision, pages 111–126, 2014. 3
    Google ScholarLocate open access versionFindings
  • A. Vedaldi and K. Lenc. MatConvNet: Convolutional neural networks for matlab. In ACM Conference on Multimedia Conference, pages 689–692, 2015. 6
    Google ScholarLocate open access versionFindings
  • Waifu2x. Image super-resolution for anime-style art using deep convolutional neural networks. http://waifu2x.udp.jp/.8
    Findings
  • Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. 6
    Google ScholarLocate open access versionFindings
  • C.-Y. Yang, C. Ma, and M.-H. Yang. Single-image superresolution: A benchmark. In European Conference on Computer Vision, pages 372–386, 2014. 1, 3
    Google ScholarLocate open access versionFindings
  • J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image superresolution via sparse representation. IEEE Transactions on Image Processing, 19(11):2861–2873, 2010. 1, 3
    Google ScholarLocate open access versionFindings
  • W. Yang, J. Feng, J. Yang, F. Zhao, J. Liu, Z. Guo, and S. Yan. Deep edge guided recurrent residual learning for image super-resolution. IEEE Transactions on Image Processing, 26(12):5895–5907, 2017. 2
    Google ScholarLocate open access versionFindings
  • R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pages 711–730, 2010. 7
    Google ScholarLocate open access versionFindings
  • K. Zhang, X. Zhou, H. Zhang, and W. Zuo. Revisiting single image super-resolution under internet environment: blur kernels and reconstruction algorithms. In Pacific Rim Conference on Multimedia, pages 677–687, 2015. 3
    Google ScholarLocate open access versionFindings
  • K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, pages 3142–3155, 2017. 2, 4, 6, 7
    Google ScholarLocate open access versionFindings
  • K. Zhang, W. Zuo, S. Gu, and L. Zhang. Learning deep CNN denoiser prior for image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3929– 3938, July 2017. 1, 2, 3, 6, 7
    Google ScholarLocate open access versionFindings
  • Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 2
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments