AI帮你理解科学

AI 生成解读视频

AI抽取解析论文重点内容自动生成视频


pub
生成解读视频

AI 溯源

AI解析本论文相关学术脉络


Master Reading Tree
生成 溯源树

AI 精读

AI抽取本论文的概要总结


微博一下
We proposed a novel regularizer in this paper to guide the discriminator in Generative Adversarial Networks to better allocate its model capacity

Improving GAN Training via Binarized Representation Entropy (BRE) Regularization.

ICLR, (2018)

被引用17|浏览33
EI
下载 PDF 全文
引用
微博一下

摘要

We propose a novel regularizer to improve the training of Generative Adversarial Networks (GANs). The motivation is that when the discriminator D spreads out its model capacity in the right way, the learning signals given to the generator G are more informative and diverse, which helps G to explore better and discover the real data manifo...更多

代码

数据

0
简介
  • Generative Adversarial Network (GAN) (Goodfellow et al, 2014) has been a new promising approach to unsupervised learning of complex high dimensional data in the last two years, with successful applications on image data (Isola et al, 2016; Shrivastava et al, 2016), and high potential for predictive representation learning (Mathieu et al, 2015) as well as reinforcement learning (Finn et al, 2016; Henderson et al, 2017).
  • Despite its success in generating high-quality data, such adversarial game setting raises challenges for the training of GANs. Many architectures and techniques have been proposed (Radford et al, 2015; Salimans et al, 2016; Gulrajani et al, 2017) to reduce extreme failures and improve the sample quality of generated data.
  • Arora et al (2017) showed that the capacity of D plays an essential role in giving G sufficient learning guidances to model the complex real data distribution.
  • D could fail to distinguish real and generated data distributions even when their Jensen-Shannon divergence or Wasserstein distance is not small
重点内容
  • Generative Adversarial Network (GAN) (Goodfellow et al, 2014) has been a new promising approach to unsupervised learning of complex high dimensional data in the last two years, with successful applications on image data (Isola et al, 2016; Shrivastava et al, 2016), and high potential for predictive representation learning (Mathieu et al, 2015) as well as reinforcement learning (Finn et al, 2016; Henderson et al, 2017)
  • Generative Adversarial Networks learn from unlabeled data by engaging the generative model (G ) in an adversarial game with a discriminator (D )
  • Many theoretical and practical open problems still remain, which have impeded the ease-of-use of Generative Adversarial Networks in new problems
  • We propose a novel regularizer to guide D to have a better model capacity allocation
  • Using a 2D synthetic dataset and CIFAR10 dataset (Krizhevsky, 2009), we show that our binarized representation entropy regularizer improves unsupervised Generative Adversarial Networks training in two ways: (a) when Generative Adversarial Networks training is unstable (for e.g. due to architectures that are less well tuned than DCGAN (Radford et al, 2015)), binarized representation entropy regularizer stabilizes the training and achieves much-improved results, often surpassing tuned configurations. (b) with architecture and hyperparameters settings that are engineered to be stable already, binarized representation entropy regularizer makes Generative Adversarial Networks learning converges faster
  • We proposed a novel regularizer in this paper to guide the discriminator in Generative Adversarial Networks to better allocate its model capacity
方法
  • Using a 2D synthetic dataset and CIFAR10 dataset (Krizhevsky, 2009), the authors show that the BRE improves unsupervised GAN training in two ways: (a) when GAN training is unstable (for e.g. due to architectures that are less well tuned than DCGAN (Radford et al, 2015)), BRE stabilizes the training and achieves much-improved results, often surpassing tuned configurations. (b) with architecture and hyperparameters settings that are engineered to be stable already, BRE makes GAN learning converges faster.
  • (b) with architecture and hyperparameters settings that are engineered to be stable already, BRE makes GAN learning converges faster.
  • The authors demonstrate that BRE regularization improves semi-supervised classification accuracy on CIFAR10 and SVHN dataset (Netzer et al, 2011).
  • The first column shows real data points and generated data points.
  • The last column shows the probability control iter:0 iter:20000 iter:4000 control control treat iter:0 iter:20000 iter:4000 treat treat
结论
  • DISCUSSION AND FUTURE

    WORK

    There are still many unexplored avenues along this line of research.
  • On networks with tanh, input gradient diversity regularizer with either cosine similarity or a soft-sign based regularizer like BRE does not work.
  • The authors performed some preliminary experiments, and found that reconstructing real data as auxilary tasks worsens the resulting learned G.
  • Based on the relation between the model capacity and the activation pattern of the network, the authors constructed the regularizer to encourage a high joint entropy of the activation pattern on the hidden layers of the discriminator D.
  • Experimental results demonstrated the benefits of the new regularizer: faster progress in the initial phase of learning thanks to improved exploration, more stable convergence, and better final results in both unsupervised and semi-supervised learning
表格
  • Table1: BRE on various architectures: no-BRE is the baseline in each case; with BRE weight in other cases is set to 1.; single and multi signify whether BRE is applied on one layer in the middle of D or multiple (see Appendix A for more details); ln for layer normalization in G and D (default is batchnorm); tanh means the softsign nonlinearity in BRE is replaced by tanh
  • Table2: Semi supervised learning on CIFAR10: feature matching (FM) from <a class="ref-link" id="cSalimans_et+al_2016_a" href="#rSalimans_et+al_2016_a">Salimans et al (2016</a>)); 1000 labeled training examples
  • Table3: Reconstruction as an auxiliary task worsens results. λrecon is the weight of the l2 reconstruction loss term. With both batch or layer normalization, reconstruction auxiliary task hurts the final results
Download tables as Excel
相关工作
  • Other regularization/gradient penalty techniques have also been proposed to stabilize GANs training (Gulrajani et al, 2017; Nagarajan & Kolter, 2017) recently. Gulrajani et al (2017) adds an input gradient penalty to the update of D , so that the magnitude of signals passed to G is controlled. Nagarajan & Kolter (2017) modifies the update of G to avoid going where the magnitude of ∇xD(x) is large. These methods, as well as other similar works that constrain the input gradient norm or the Lipschitz constant of D , all try to stabilize the training dynamics by regularizing the learning signal magnitude. This is different from our method that diversifies the learning signal directions. As discussed in the previous section, the diversified signal directions help both the convergence speed and the stability of the training. In Sec. 4.2, we empirically demonstrate that our proposed method achieves better results than Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al, 2017).
基金
  • Proposes a novel regularizer to improve the training of Generative Adversarial Networks
  • Demonstrates that even with sufficient maximum capacity, D might not allocate its capacity in a desirable way that facilitates convergence to a good equilibrium
  • Proposes a novel regularizer to guide D to have a better model capacity allocation
  • Proposes a new regularizer for training GANs where D is a rectifier net
  • Explores the question on where and how D can utilize its limited capacity effectively for better training convergence
引用论文
  • Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). arXiv preprint arXiv:1703.00573, 2017.
    Findings
  • David Berthelot, Tom Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717, 2017.
    Findings
  • Georges A Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4):1315–1321, 1999.
    Google ScholarLocate open access versionFindings
  • Jeff Donahue, Philipp Krahenbuhl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782, 2016.
    Findings
  • Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606.00704, 2016.
    Findings
  • Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852, 2016.
    Findings
  • Dmitry Gavinsky and Pavel Pudlak. On the joint entropy of d-wise-independent variables. arXiv preprint arXiv:1503.08154, 2015.
    Findings
  • Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014.
    Google ScholarLocate open access versionFindings
  • Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. arXiv preprint arXiv:1704.00028, 2017.
    Findings
  • Leonard Hasenclever, Stefan Webb, Thibaut Lienart, Sebastian Vollmer, Balaji Lakshminarayanan, Charles Blundell, and Yee Whye Teh. Distributed bayesian learning with stochastic natural gradient expectation propagation and the posterior server. Journal of Machine Learning Research, 18(106):1–37, 2017.
    Google ScholarLocate open access versionFindings
  • Peter Henderson, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. arXiv preprint arXiv:1709.06683, 2017.
    Findings
  • Jose Miguel Hernandez-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Learning, pp. 1861– 1869, 2015.
    Google ScholarLocate open access versionFindings
  • Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.
    Findings
  • Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004, 2016.
    Findings
  • Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
    Google ScholarFindings
  • Jan Kybic and Ivan Vnucko. Approximate all nearest neighbor search for high dimensional entropy estimation for image registration. Signal Processing, 92(5):1302–1316, 2012.
    Google ScholarLocate open access versionFindings
  • Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738, 2015.
    Google ScholarLocate open access versionFindings
  • Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579–2605, 2008.
    Google ScholarLocate open access versionFindings
  • Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440, 2015.
    Findings
  • Erik G Miller. A new class of entropy estimators for multi-dimensional densities. In Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03). 2003 IEEE International Conference on, volume 3, pp. III–297. IEEE, 2003.
    Google ScholarLocate open access versionFindings
  • Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in neural information processing systems, pp. 2924–2932, 2014.
    Google ScholarLocate open access versionFindings
  • Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. arXiv preprint arXiv:1706.04156, 2017.
    Findings
  • Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5, 2011.
    Google ScholarLocate open access versionFindings
  • Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
    Findings
  • Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. arXiv preprint arXiv:1606.05336, 2016.
    Findings
  • Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234–2242, 2016.
    Google ScholarLocate open access versionFindings
  • David W Scott. Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, 2015.
    Google ScholarFindings
  • Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training. arXiv preprint arXiv:1612.07828, 2016.
    Findings
  • Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126, 2016.
    Findings
您的评分 :
0

 

标签
评论
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科