AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We introduce Noise2Same, a self-supervised framework for deep image denoising

Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising

NIPS 2020, (2020)

Cited by: 0|Views367
EI
Full Text
Bibtex
Weibo

Abstract

Self-supervised frameworks that learn denoising models with merely individual noisy images have shown strong capability and promising performance in various image denoising tasks. Existing self-supervised denoising frameworks are mostly built upon the same theoretical foundation, where the denoising models are required to be J-invariant...More
Introduction
  • The quality of deep learning methods for signal reconstruction from noisy images, known as deep image denoising, has benefited from the advanced neural network architectures such as ResNet [8], U-Net [19] and their variants [29, 16, 26, 31, 25, 14].
  • When neither clean images nor paired noisy images are available, various self-supervised denoising methods have been developed [10, 1, 12] by assuming that the noise is zero-mean and independent among all dimensions.
  • These methods are trained on individual noisy images to minimize the self-supervised loss L(f ) = Ex f (x) − x 2.
  • In order to prevent the self-supervised training from collapsing into leaning the identity function, Batson et al [1] point out that the denoising function f should be J -invariant, as defined below
Highlights
  • The quality of deep learning methods for signal reconstruction from noisy images, known as deep image denoising, has benefited from the advanced neural network architectures such as ResNet [8], U-Net [19] and their variants [29, 16, 26, 31, 25, 14]
  • When neither clean images nor paired noisy images are available, various self-supervised denoising methods have been developed [10, 1, 12] by assuming that the noise is zero-mean and independent among all dimensions. These methods are trained on individual noisy images to minimize the self-supervised loss L(f ) = Ex f (x) − x 2
  • We introduce Noise2Same, a self-supervised framework for deep image denoising
  • The most direct application of Noise2Same is to perform denoising on digital images captured under poor conditions
  • In addition to image denoising applications, the self-supervised denoising framework could be extended to other domains such as audio noise reduction and single-cell [1]
  • As many imaging-based research tasks and computer vision applications may be built upon the denoising algorithms, the failure of Noise2Same could potentially lead to biases or failures in these tasks and applications
Methods
  • Datasets ImageNet HànZì Planaria BSD68 Traditional.
  • Input NLM [3] BM3D [5].
  • 8.41 25.80 / 24.03 / 21.62 Supervised.
  • Noise2True Noise2Noise [13].
Conclusion
  • Conclusion and Future Work

    The authors analyzed the existing blind-spot-based denoising methods and introduced Noise2Same, a novel self-supervised denoising method, which removes the assumption and over-restriction on the neural network as a J -invariant function.
  • The combination of self-supervised denoising result and the noise model has be shown to provide additional performance gain.
  • As Noise2Same does not need paired clean data, paired noisy data, nor the noise model, its application scenarios could be much broader than both traditional supervised and existing self-supervised denoising frameworks.
  • In addition to image denoising applications, the self-supervised denoising framework could be extended to other domains such as audio noise reduction and single-cell [1].
  • As many imaging-based research tasks and computer vision applications may be built upon the denoising algorithms, the failure of Noise2Same could potentially lead to biases or failures in these tasks and applications
Summary
  • Introduction:

    The quality of deep learning methods for signal reconstruction from noisy images, known as deep image denoising, has benefited from the advanced neural network architectures such as ResNet [8], U-Net [19] and their variants [29, 16, 26, 31, 25, 14].
  • When neither clean images nor paired noisy images are available, various self-supervised denoising methods have been developed [10, 1, 12] by assuming that the noise is zero-mean and independent among all dimensions.
  • These methods are trained on individual noisy images to minimize the self-supervised loss L(f ) = Ex f (x) − x 2.
  • In order to prevent the self-supervised training from collapsing into leaning the identity function, Batson et al [1] point out that the denoising function f should be J -invariant, as defined below
  • Methods:

    Datasets ImageNet HànZì Planaria BSD68 Traditional.
  • Input NLM [3] BM3D [5].
  • 8.41 25.80 / 24.03 / 21.62 Supervised.
  • Noise2True Noise2Noise [13].
  • Conclusion:

    Conclusion and Future Work

    The authors analyzed the existing blind-spot-based denoising methods and introduced Noise2Same, a novel self-supervised denoising method, which removes the assumption and over-restriction on the neural network as a J -invariant function.
  • The combination of self-supervised denoising result and the noise model has be shown to provide additional performance gain.
  • As Noise2Same does not need paired clean data, paired noisy data, nor the noise model, its application scenarios could be much broader than both traditional supervised and existing self-supervised denoising frameworks.
  • In addition to image denoising applications, the self-supervised denoising framework could be extended to other domains such as audio noise reduction and single-cell [1].
  • As many imaging-based research tasks and computer vision applications may be built upon the denoising algorithms, the failure of Noise2Same could potentially lead to biases or failures in these tasks and applications
Tables
  • Table1: D(f ) and PSNR of f trained through mask-based blind-spot methods with different replacement strategies on BSD68. The last column corresponds to a strictly J -invariant model
  • Table2: D(f ) and PSNR of f on trained through masktween f trained with different replacement based blind-spot methods with the same replacement strategies on the BSD68 dataset [<a class="ref-link" id="c15" href="#r15">15</a>]. We strategy on different datasets
  • Table3: Comparisons among denoising methods on different datasets, in terms of Peak Signal-toNoise Ratio (PSNR). The post-processing of Laine et al [<a class="ref-link" id="c12" href="#r12">12</a>] that requires information about the noise model is included under the Self-Supervised + noise model category and is excluded under the Self-Supervised category. Noise2Self-Noise and Noise2Self-Donut refer to two masking strategies mentioned in [<a class="ref-link" id="c1" href="#r1">1</a>], where the original results presented in [<a class="ref-link" id="c1" href="#r1">1</a>] are produced by the noise masking. Bold numbers indicate the best performance among self-supervised methods
Download tables as Excel
Funding
  • Acknowledgments and Disclosure of Funding This work was supported in part by National Science Foundation grant DBI-2028361
Study subjects and analysis
datasets: 4
Signal-to-Noise Ratio (PSNR) as the evaluation metric. The comparison results between our Noise2Same and the baselines in terms of PSNR on the four datasets are summarized in Table 3 and visualized in Figure 2 and Appendix F. The results show that our Noise2Same achieve remarkable improvements over previous self-supervised baselines on Im-

datasets: 4
Noise2Noise Noise2True Ground Truth. We evaluate our Noise2Same on four datasets, including RGB natural images (ImageNet ILSVRC 2012 Val [21]), generated hand-written Chinese character images (HànZì [1]), physically captured 3D microscopy data (Planaria [27]) and grey-scale natural images (BSD68 [15]). The four datasets have different noise types and levels

datasets: 4
We evaluate our Noise2Same on four datasets, including RGB natural images (ImageNet ILSVRC 2012 Val [21]), generated hand-written Chinese character images (HànZì [1]), physically captured 3D microscopy data (Planaria [27]) and grey-scale natural images (BSD68 [15]). The four datasets have different noise types and levels. The constructions of the four datasets are described in Appendix C

datasets: 4
The four datasets have different noise types and levels. The constructions of the four datasets are described in Appendix C. 5.1 Comparisons with Baselines

Reference
  • Joshua Batson and Loïc Royer. Noise2self: Blind denoising by self-supervision. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 524–533, 2019.
    Google ScholarLocate open access versionFindings
  • Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11036–11045, 2019.
    Google ScholarLocate open access versionFindings
  • Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 60–65. IEEE, 2005.
    Google ScholarLocate open access versionFindings
  • Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE transactions on pattern analysis and machine intelligence, 39(6):1256–1272, 2016.
    Google ScholarLocate open access versionFindings
  • Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising with block-matching and 3d filtering. In Image Processing: Algorithms and Systems, Neural Networks, and Machine Learning, volume 6064, page 606414. International Society for Optics and Photonics, 2006.
    Google ScholarLocate open access versionFindings
  • Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
    Google ScholarLocate open access versionFindings
  • Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1712–1722, 2019.
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
    Google ScholarLocate open access versionFindings
  • Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pages 5574–5584, 2017.
    Google ScholarLocate open access versionFindings
  • Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2129–2137, 2019.
    Google ScholarLocate open access versionFindings
  • Alexander Krull, Tomas Vicar, and Florian Jug. Probabilistic noise2void: Unsupervised contentaware denoising. arXiv preprint arXiv:1906.00651, 2019.
    Findings
  • Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In Advances in Neural Information Processing Systems, pages 6968–6978, 2019.
    Google ScholarLocate open access versionFindings
  • Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, Timo Aila, et al. Noise2noise. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2971–2980, 2018.
    Google ScholarLocate open access versionFindings
  • Yi Liu, Hao Yuan, Zhengyang Wang, and Shuiwang Ji. Global pixel transformers for virtual staining of microscopy images. IEEE Transactions on Medical Imaging, 39(6):2256–2266, 2020.
    Google ScholarLocate open access versionFindings
  • D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 8th IEEE International Conference on Computer Vision, volume 2, pages 416–423, July 2001.
    Google ScholarLocate open access versionFindings
  • Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV), pages 565–571. IEEE, 2016.
    Google ScholarLocate open access versionFindings
  • Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. arXiv preprint arXiv:1910.11908, 2019.
    Findings
  • Mangal Prakash, Tim-Oliver Buchholz, Manan Lalit, Pavel Tomancak, Florian Jug, and Alexander Krull. Leveraging self-supervised denoising for image segmentation. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 428–432. IEEE, 2020.
    Google ScholarLocate open access versionFindings
  • Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241.
    Google ScholarLocate open access versionFindings
  • Stefan Roth and Michael J Black. Fields of experts. International Journal of Computer Vision, 82(2):205, 2009.
    Google ScholarLocate open access versionFindings
  • Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
    Google ScholarLocate open access versionFindings
  • Uwe Schmidt and Stefan Roth. Shrinkage fields for effective image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2774–2781. IEEE Computer Society, 2014.
    Google ScholarLocate open access versionFindings
  • Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446–9454, 2018.
    Google ScholarLocate open access versionFindings
  • Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(Dec):3371–3408, 2010.
    Google ScholarLocate open access versionFindings
  • Zhengyang Wang, Yaochen Xie, and Shuiwang Ji. Global voxel transformer networks for augmented microscopy. arXiv preprint arXiv:2008.02340, 2020.
    Findings
  • Zhengyang Wang, Na Zou, Dinggang Shen, and Shuiwang Ji. Non-local u-nets for biomedical image segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6315–6322, 2020.
    Google ScholarLocate open access versionFindings
  • Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, et al. Contentaware image restoration: pushing the limits of fluorescence microscopy. Nature methods, 15(12):1090–1097, 2018.
    Google ScholarLocate open access versionFindings
  • Jun Xu, Yuan Huang, Li Liu, Fan Zhu, Xingsong Hou, and Ling Shao. Noisy-as-clean: learning unsupervised denoising from the corrupted image. arXiv preprint arXiv:1906.06878, 2019.
    Findings
  • Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
    Google ScholarLocate open access versionFindings
  • Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058–1067, 2017.
    Google ScholarLocate open access versionFindings
  • Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
    Google ScholarLocate open access versionFindings
Author
Yaochen Xie
Yaochen Xie
Your rating :
0

 

Tags
Comments
小科