Learning a Single Convolutional Super-Resolution Network for Multiple Degradations
computer vision and pattern recognition, 2018.
EI
Weibo:
Abstract:
Recent years have witnessed the unprecedented success of deep convolutional neural networks (CNNs) in single image super-resolution (SISR). However, existing CNN-based SISR methods mostly assume that a low-resolution (LR) image is bicubicly downsampled from a high-resolution (HR) image, thus inevitably giving rise to poor performance when...More
Code:
Data:
Introduction
- Single image super-resolution (SISR) aims to recover a high-resolution (HR) version of a low-resolution (LR) input.
- SISR methods can be broadly classified into three categories, i.e., interpolation-based methods, model-based optimization methods and discriminative learning methods.
- The integration of convolutional neural network (CNN) denoiser prior and model-based optimization can improve the efficiency to some extent, it still suffers from the typical drawbacks of model-based optimization methods, e.g., it is not in an end-to-end learning manner and involves hand-designed parameters [57].
- Discriminative learning methods have attracted considerable attentions due to their favorable SISR performance in terms of effectiveness and efficiency.
- Recent years have witnessed a dramatic upsurge of using CNN for SISR
Highlights
- Single image super-resolution (SISR) aims to recover a high-resolution (HR) version of a low-resolution (LR) input
- Where x⊗k represents the convolution between a blur kernel k and a latent HR image x, ↓s is a subsequent downsampling operation with scale factor s, and n usually is additive white Gaussian noise (AWGN) with standard deviation σ
- The integration of convolutional neural network (CNN) denoiser prior and model-based optimization can improve the efficiency to some extent, it still suffers from the typical drawbacks of model-based optimization methods, e.g., it is not in an end-to-end learning manner and involves hand-designed parameters [57]
- We proposed an effective super-resolution network with high scalability of handling multiple degradations via a single model
- Degradation maps are obtained by a simple dimensionality stretching of the degradation parameters
- The results on real LR images showed that the proposed method can reconstruct visually plausible HR images
Methods
- Before solving the problem of SISR, it is important to have a clear understanding of the degradation model which is not limited to Eqn (1).
- Another practical degradation model can be given by y = (x ↓s) ⊗ k + n.
- When ↓ is the bicubic downsampler, Eqn (2) corresponds to a deblurring problem followed by a SISR problem with bicubic degradation.
- The authors make a short discussion on blur kernel k, noise n and downsampler ↓
Conclusion
- The authors proposed an effective super-resolution network with high scalability of handling multiple degradations via a single model.
- Different from existing CNNbased SISR methods, the proposed super-resolver takes both LR image and its degradation maps as input.
- The results on synthetic LR images demonstrated that the proposed super-resolver can produce state-of-the-art results on bicubic degradation and perform favorably on other degradations and even spatially variant degradations.
- The proposed super-resolver offers a feasible solution toward practical CNN-based SISR applications
Summary
Introduction:
Single image super-resolution (SISR) aims to recover a high-resolution (HR) version of a low-resolution (LR) input.- SISR methods can be broadly classified into three categories, i.e., interpolation-based methods, model-based optimization methods and discriminative learning methods.
- The integration of convolutional neural network (CNN) denoiser prior and model-based optimization can improve the efficiency to some extent, it still suffers from the typical drawbacks of model-based optimization methods, e.g., it is not in an end-to-end learning manner and involves hand-designed parameters [57].
- Discriminative learning methods have attracted considerable attentions due to their favorable SISR performance in terms of effectiveness and efficiency.
- Recent years have witnessed a dramatic upsurge of using CNN for SISR
Objectives:
It is natural to raise the following questions, which are the focus of the paper: (i) Can the authors learn a single model to effectively handle multiple and even spatially variant degradations? (ii) Is it possible to use synthetic data to train a model with high practicability? This work aims to make one of the first attempts towards answering these two questions.- It is natural to raise the following questions, which are the focus of the paper: (i) Can the authors learn a single model to effectively handle multiple and even spatially variant degradations?
- (ii) Is it possible to use synthetic data to train a model with high practicability?
- This work aims to make one of the first attempts towards answering these two questions.
- Instead of handling the bicubic degradation only, the aim is to learn a single network to handle multiple degradations
Methods:
Before solving the problem of SISR, it is important to have a clear understanding of the degradation model which is not limited to Eqn (1).- Another practical degradation model can be given by y = (x ↓s) ⊗ k + n.
- When ↓ is the bicubic downsampler, Eqn (2) corresponds to a deblurring problem followed by a SISR problem with bicubic degradation.
- The authors make a short discussion on blur kernel k, noise n and downsampler ↓
Conclusion:
The authors proposed an effective super-resolution network with high scalability of handling multiple degradations via a single model.- Different from existing CNNbased SISR methods, the proposed super-resolver takes both LR image and its degradation maps as input.
- The results on synthetic LR images demonstrated that the proposed super-resolver can produce state-of-the-art results on bicubic degradation and perform favorably on other degradations and even spatially variant degradations.
- The proposed super-resolver offers a feasible solution toward practical CNN-based SISR applications
Tables
- Table1: Average PSNR and SSIM results for bicubic degradation on datasets Set5 [<a class="ref-link" id="c3" href="#r3">3</a>], Set14 [<a class="ref-link" id="c54" href="#r54">54</a>], BSD100 [<a class="ref-link" id="c33" href="#r33">33</a>] and Urban100 [<a class="ref-link" id="c19" href="#r19">19</a>]. The best two results are highlighted in red and blue colors, respectively
- Table2: Average PSNR and SSIM results of different methods with different degradations on Set5. The best results are highlighted in red color. The results highlighted in gray color indicate unfair comparison due to mismatched degradation assumption
Related work
- The first work of using CNN to solve SISR can be traced back to [8] where a three-layer super-resolution network (SRCNN) was proposed. In the extended work [9], the authors investigated the impact of depth on super-resolution and empirically showed that the difficulty of training deeper model hinders the performance improvement of CNN super-resolvers. To overcome the training difficulty, Kim et al [24] proposed a very deep super-resolution (VDSR) method with residual learning strategy. Interestingly, they showed that VDSR can handle multiple scales superresolution. By analyzing the relation between CNN and MAP inference, Zhang et al [56] pointed out that CNN mainly model the prior information and they empirically demonstrated that a single model can handle multiple scales super-resolution, image deblocking and image denoising. While achieving good performance, the above methods take the bicubicly interpolated LR image as input, which not only suffers from high computational cost but also hinders the effective expansion of receptive field.
Funding
- This work is supported by National Natural Science Foundation of China (grant no. 61671182, 61471146), HK RGC General Research Fund (PolyU 152240/15E) and PolyU-Alibaba Collaborative Research Project “Quality Enhancement of Surveillance Images and Videos”
- We gratefully acknowledge the support from NVIDIA Corporation for providing us the Titan Xp GPU used in this research
Study subjects and analysis
widely-used datasets: 4
However, in order to show the advantage of the dimensionality stretching strategy, the proposed method is also compared with other CNN-based methods specifically designed for bicubic degradation. Table 1 shows the PSNR and SSIM [50] results of stateof-the-art CNN-based SISR methods on four widely-used datasets. As one can see, SRMD achieves comparable results with VDSR at small scale factor and outperforms VDSR at large scale factor
Reference
- E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, volume 3, pages 126–135, July 2017. 6
- S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167–1183, 2001
- M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. A. Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference, 2012. 7
- S. A. Bigdeli, M. Jin, P. Favaro, and M. Zwicker. Deep meanshift priors for image restoration. In Advances in Neural Information Processing Systems, 2017. 1, 2
- G. Boracchi and A. Foi. Modeling the performance of image restoration from motion blur. IEEE Transactions on Image Processing, 21(8):3502–3517, Aug 2012. 3
- Y. Chen, W. Yu, and T. Pock. On learning optimized reaction diffusion processes for effective image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5261–5269, 2015. 2
- Z. Cui, H. Chang, S. Shan, B. Zhong, and X. Chen. Deep network cascade for image super-resolution. In European Conference on Computer Vision, pages 49–64, 2014. 3
- C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184–199, 2014. 2
- C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2016. 2, 7
- C. Dong, C. C. Loy, and X. Tang. Accelerating the superresolution convolutional neural network. In European Conference on Computer Vision, pages 391–407, 2016. 2
- W. Dong, L. Zhang, G. Shi, and X. Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 22(4):1620–1630, 2013. 1, 2, 3, 6, 7
- N. Efrat, D. Glasner, A. Apartsin, B. Nadler, and A. Levin. Accurate blur models vs. image priors in single image superresolution. In IEEE International Conference on Computer Vision, pages 2832–2839, 2013. 1, 3
- K. Egiazarian and V. Katkovnik. Single image superresolution via BM3D sparse coding. In European Signal Processing Conference, pages 2849–2853, 2015. 1
- W. Freeman and C. Liu. Markov random fields for superresolution and texture synthesis. Advances in Markov Random Fields for Vision and Image Processing, 1:155–165, 2011. 3
- D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In IEEE International Conference on Computer Vision, pages 349–356, 2009. 3
- I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014. 1, 2
- H. He and W.-C. Siu. Single image super-resolution using Gaussian process regression. In IEEE Conference on Computer Vision and Pattern Recognition, pages 449–456, 2011. 3
- K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 1
- J.-B. Huang, A. Singh, and N. Ahuja. Single image superresolution from transformed self-exemplars. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2015. 7, 8
- S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448– 456, 2015. 4, 5
- M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Advances in Neural Information Processing Systems, pages 2017–2025, 2015. 1, 5
- J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694–711, 2016. 2
- J. Kim, J. Kwon Lee, and K. Mu Lee. Deeply-recursive convolutional network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1637–1645, 2016. 2
- J. Kim, J. K. Lee, and K. M. Lee. Accurate image superresolution using very deep convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1646–1654, 2016. 2, 4, 7, 8
- D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015. 5, 6
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012. 4, 5, 6
- W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate superresolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 624–632, July 2017. 2, 7
- Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, 2015. 1
- C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4681–4690, July 2017. 2, 5, 7
- B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 136–144, July 2017. 2
- K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing, 26(2):1004–1016, 2017. 6
- J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2272–2279, 2009. 1
- D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference on Computer Vision, volume 2, pages 416–423, July 2001. 6, 7
- T. Meinhardt, M. Moller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In IEEE International Conference on Computer Vision, pages 1781–1790, 2017. 2
- J. Pan, Z. Hu, Z. Su, and M.-H. Yang. Deblurring text images via L0-regularized intensity and gradient prior. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2908, 2014. 8
- T. Peleg and M. Elad. A statistical prediction model based on sparse representations for single image super-resolution. IEEE transactions on Image Processing, 23(6):2569–2582, 2014. 3
- J. S. Ren, L. Xu, Q. Yan, and W. Sun. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems, pages 901–909, 2015. 2
- G. Riegler, S. Schulter, M. Ruther, and H. Bischof. Conditioned regression models for non-blind single image superresolution. In IEEE International Conference on Computer Vision, pages 522–530, 2015. 3, 5
- Y. Romano, M. Elad, and P. Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4):1804–1844, 2017. 2
- Y. Romano, J. Isidoro, and P. Milanfar. RAISR: rapid and accurate image super resolution. IEEE Transactions on Computational Imaging, 3(1):110–125, 2017. 1
- W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1874–1883, 2016. 2, 4
- Y. Shi, K. Wang, C. Chen, L. Xu, and L. Lin. Structurepreserving image super-resolution via contextualized multitask learning. IEEE Transactions on Multimedia, 2017. 2
- A. Singh, F. Porikli, and N. Ahuja. Super-resolving noisy images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2846–2853, 2014. 3
- Y. Tai, J. Yang, and X. Liu. Image super-resolution via deep recursive residual network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3147–3155, 2017. 2, 7
- Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In IEEE International Conference on Computer Vision, pages 4539–4547, 2017. 2
- R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang. Ntire 2017 challenge on single image superresolution: Methods and results. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 114–125, July 2017. 2
- R. Timofte, V. De Smet, and L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian Conference on Computer Vision, pages 111–126, 2014. 3
- A. Vedaldi and K. Lenc. MatConvNet: Convolutional neural networks for matlab. In ACM Conference on Multimedia Conference, pages 689–692, 2015. 6
- Waifu2x. Image super-resolution for anime-style art using deep convolutional neural networks. http://waifu2x.udp.jp/.8
- Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. 6
- C.-Y. Yang, C. Ma, and M.-H. Yang. Single-image superresolution: A benchmark. In European Conference on Computer Vision, pages 372–386, 2014. 1, 3
- J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image superresolution via sparse representation. IEEE Transactions on Image Processing, 19(11):2861–2873, 2010. 1, 3
- W. Yang, J. Feng, J. Yang, F. Zhao, J. Liu, Z. Guo, and S. Yan. Deep edge guided recurrent residual learning for image super-resolution. IEEE Transactions on Image Processing, 26(12):5895–5907, 2017. 2
- R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pages 711–730, 2010. 7
- K. Zhang, X. Zhou, H. Zhang, and W. Zuo. Revisiting single image super-resolution under internet environment: blur kernels and reconstruction algorithms. In Pacific Rim Conference on Multimedia, pages 677–687, 2015. 3
- K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, pages 3142–3155, 2017. 2, 4, 6, 7
- K. Zhang, W. Zuo, S. Gu, and L. Zhang. Learning deep CNN denoiser prior for image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3929– 3938, July 2017. 1, 2, 3, 6, 7
- Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 2
Full Text
Tags
Comments