Resprop: Reuse Sparsified Backpropagation
2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)(2020)
摘要
The success of Convolutional Neural Networks (CNNs) in various applications is accompanied by a significant increase in computation and training time. In this work, we focus on accelerating training by observing that about 90% of gradients are reusable during training. Leveraging this observation, we propose a new algorithm, ReuseSparse-Backprop (ReSprop), as a method to sparsity gradient vectors during CNN training. ReSprop maintains stateof-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets with less than 1.1% accuracy loss while enabling a reduction in back-propagation computations by a factor of 10x resulting in a 2.7x overall speedup in training. As the computation reduction introduced by ReSprop is accomplished by introducing fine-grained sparsity that reduces computation efficiency on GPUs, we introduce a generic sparse convolution neural network accelerator (GSCN), which is designed to accelerate sparse back-propagation convolutions. When combined with ReSprop, GSCN achieves 8.0x and 7.2x speedup in the backward pass on ResNet34 and VGG16 versus a GTX 1080 Ti GPU.
更多查看译文
关键词
back-propagation convolutions,ReSprop,convolutional neural networks,Reuse-Sparse-Backprop,gradient vectors,CNN training,sparse convolution neural network accelerator
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络