Can We Gain More from Orthogonality Regularizations in Training Deep Networks?
neural information processing systems(2018)
摘要
This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how we can enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on three state-of-the-art models: ResNet, WideResNet, and ResNeXt, on CIFAR-10 and CIFAR-100 datasets. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and accelerated and more stable convergences.
更多查看译文
关键词
restricted isometry property,mutual coherence
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络