AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We train a neural network that learns to find binary values for weights, which reduces the size of network by ∼32× and provide the possibility of loading very deep neural networks into portable devices with limited memory

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks.

ECCV, (2016)

Cited by: 3376|Views618
EI
Full Text
Bibtex
Weibo

Abstract

We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32(times ) memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks ap...More

Code:

Data:

0
Introduction
  • Deep neural networks (DNN) have shown significant improvements in several application domains including computer vision and speech recognition.
  • Concurrent to the recent progress in recognition, interesting advancements have been happening in virtual reality (VR by Oculus) [8], augmented reality (AR by HoloLens) [9], and smart wearable devices
  • Putting these two pieces together, the authors argue that it is the right time to equip smart portable devices with the power of state-of-the-art recognition systems.
Highlights
  • Deep neural networks (DNN) have shown significant improvements in several application domains including computer vision and speech recognition
  • Convolutional neural networks show reliable results on object recognition and detection that are useful in real world applications
  • Our experimental results show that our proposed method for binarizing convolutional neural networks outperforms the state-of-the-art network binarization method of [11] by a large margin (16.3 %) on top-1 image classification in the ImageNet challenge ILSVRC2012
  • Our contribution is two-fold: First, we introduce a new way of binarizing the weight values in convolutional neural networks and show the advantage of our solution compared to state-of-the-art solutions
  • Efficient, and accurate binary approximations for neural networks
  • We train a neural network that learns to find binary values for weights, which reduces the size of network by ∼32× and provide the possibility of loading very deep neural networks into portable devices with limited memory
Methods
  • The authors evaluate the method by analyzing its efficiency and accuracy.
  • The authors measure the efficiency by computing the computational speedup achieved by the binary convolution vs standard convolution.
  • The authors perform image classification on the large-scale ImageNet dataset.
  • This paper is the first work that evaluates binary neural networks on the ImageNet dataset.
  • The authors compare the method with two recent works on binarizing neural networks; BinaryConnect [38] and BinaryNet [11].
Results
  • The authors compare the method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than 16 % in top-1 accuracy.
  • The authors' experimental results show that the proposed method for binarizing convolutional neural networks outperforms the state-of-the-art network binarization method of [11] by a large margin (16.3 %) on top-1 image classification in the ImageNet challenge ILSVRC2012.
  • Removing β reduces the accuracy by a small margin
Conclusion
  • Efficient, and accurate binary approximations for neural networks.
  • The authors propose an architecture, XNOR-Net, that uses mostly bitwise operations to approximate convolutions.
  • This provides ∼58× speed up and enables the possibility of running the inference of state of the art deep neural network on CPU in real-time
Tables
  • Table1: This table compares the final accuracies (Top1 - Top5) of the full precision network with our binary precision networks; Binary-Weight-Networks (BWN) and XNOR-Networks (XNOR-Net) and the competitor methods; BinaryConnect (BC) and BinaryNet (BNN)
  • Table2: This table compares the final classification accuracy achieved by our binary precision networks with the full precision network in ResNet-18 and GoogLenet architectures
  • Table3: In this table, we evaluate two key elements of our approach; computing the optimal scaling factors and specifying the right order for layers in a block of CNN with binary input. (a) demonstrates the importance of the scaling factor in training binary-weight-networks and (b) shows that our way of ordering the layers in a block of CNN is crucial for training XNOR-Networks. C,B,A,P stands for Convolutional, BatchNormalization, Active function (here binary activation), and Pooling respectively
Download tables as Excel
Related work
  • Deep neural networks often suffer from over-parametrization and large amounts of redundancy in their models. This typically results in inefficient computation and memory usage [12]. Several methods have been proposed to address efficient training and inference in deep neural networks.

    Shallow networks: Estimating a deep neural network with a shallower model reduces the size of a network. Early theoretical work by Cybenko shows that a network with a large enough single hidden layer of sigmoid units can approximate any decision boundary [13]. In several areas (e.g., vision and speech), however, shallow networks cannot compete with deep models [14]. [15] trains a shallow network on SIFT features to classify the ImageNet dataset. They show it is difficult to train shallow networks with large number of parameters. [16] provides empirical evidence on small datasets (e.g., CIFAR-10) that shallow nets are capable of learning the same functions as deep nets. In order to get the similar accuracy, the number of parameters in the shallow network must be close to the number of parameters in the deep network. They do this by first training a state-of-the-art deep model, and then training a shallow model to mimic the deep model. These methods are different from our approach because we use the standard deep architectures not the shallow estimations.
Funding
  • This work is in part supported by ONR N00014-13-1-0720, NSF IIS- 1338054, Allen Distinguished Investigator Award, and the Allen Institute for Artificial Intelligence
Reference
  • Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105 (2012)
    Google ScholarLocate open access versionFindings
  • Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
    Findings
  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
    Google ScholarLocate open access versionFindings
  • He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR (2015)
    Google ScholarLocate open access versionFindings
  • Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
    Google ScholarLocate open access versionFindings
  • Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)
    Google ScholarLocate open access versionFindings
  • Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)
    Google ScholarLocate open access versionFindings
  • Oculus, V.: Oculus rift-virtual reality headset for 3d gaming (2012). http://www.oculusvr.com
    Findings
  • Gottmer, M.: Merging reality and virtuality with microsoft hololens (2015)
    Google ScholarFindings
  • Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
    Google ScholarLocate open access versionFindings
  • Courbariaux, M., Bengio, Y.: Binarynet: training deep neural networks with weights and activations constrained to +1 or −1. CoRR (2016)
    Google ScholarLocate open access versionFindings
  • Denil, M., Shakibi, B., Dinh, L., de Freitas, N., et al.: Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, pp. 2148–2156 (2013)
    Google ScholarLocate open access versionFindings
  • Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Sig. Syst. 2(4), 303–314 (1989)
    Google ScholarLocate open access versionFindings
  • Seide, F., Li, G., Yu, D.: Conversational speech transcription using contextdependent deep neural networks. In: Interspeech, pp. 437–440 (2011)
    Google ScholarFindings
  • Dauphin, Y.N., Bengio, Y.: Big neural networks waste capacity. arXiv preprint arXiv:1301.3583 (2013)
    Findings
  • Ba, J., Caruana, R.: Do deep nets really need to be deep?. In: Advances in Neural Information Processing Systems, pp. 2654–2662 (2014)
    Google ScholarLocate open access versionFindings
  • Hanson, S.J., Pratt, L.Y.: Comparing biases for minimal network construction with back-propagation. In: Advances in Neural Information Processing Systems, pp. 177–185 (1989)
    Google ScholarLocate open access versionFindings
  • LeCun, Y., Denker, J.S., Solla, S.A., Howard, R.E., Jackel, L.D.: Optimal brain damage. In: NIPs, vol. 89 (1989)
    Google ScholarLocate open access versionFindings
  • Hassibi, B., Stork, D.G.: Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. Morgan Kaufmann, San Francisco (1993)
    Google ScholarFindings
  • Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)
    Google ScholarLocate open access versionFindings
  • Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Heidelberg (2015). doi:10.1007/978-3-319-24553-9 83
    Locate open access versionFindings
  • Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
    Findings
  • Chen, W., Wilson, J.T., Tyree, S., Weinberger, K.Q., Chen, Y.: Compressing neural networks with the hashing trick. arXiv preprint arXiv:1504.04788 (2015)
    Findings
  • Denton, E.L., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in Neural Information Processing Systems, pp. 1269–1277 (2014)
    Google ScholarLocate open access versionFindings
  • Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 (2014)
    Findings
  • Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
    Findings
  • Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR (2016)
    Google ScholarLocate open access versionFindings
  • Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: Squeezenet: alexnet-level accuracy with 50x fewer parameters and <1mb model size. arXiv preprint arXiv:1602.07360 (2016)
    Findings
  • Gong, Y., Liu, L., Yang, M., Bourdev, L.: Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 (2014)
    Findings
  • Arora, S., Bhaskara, A., Ge, R., Ma, T.: Provable bounds for learning some deep representations. arXiv preprint arXiv:1310.6343 (2013)
    Findings
  • Vanhoucke, V., Senior, A., Mao, M.Z.: Improving the speed of neural networks on cpus. In: Proceedings of Deep Learning and Unsupervised Feature Learning NIPS Workshop, vol. 1 (2011)
    Google ScholarLocate open access versionFindings
  • Hwang, K., Sung, W.: Fixed-point feedforward deep neural network design using weights +1, 0, and −1. In: 2014 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1–6. IEEE (2014)
    Google ScholarLocate open access versionFindings
  • Anwar, S., Hwang, K., Sung, W.: Fixed point optimization of deep convolutional neural networks for object recognition. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1131–1135. IEEE (2015)
    Google ScholarLocate open access versionFindings
  • Lin, Z., Courbariaux, M., Memisevic, R., Bengio, Y.: Neural networks with few multiplications. arXiv preprint arXiv:1510.03009 (2015)
    Findings
  • Courbariaux, M., Bengio, Y., David, J.P.: Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024 (2014)
    Findings
  • Soudry, D., Hubara, I., Meir, R.: Expectation backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights. In: Advances in Neural Information Processing Systems, pp. 963–971 (2014)
    Google ScholarLocate open access versionFindings
  • Esser, S.K., Appuswamy, R., Merolla, P., Arthur, J.V., Modha, D.S.: Backpropagation for energy-efficient neuromorphic computing. In: Advances in Neural Information Processing Systems, pp. 1117–1125 (2015)
    Google ScholarLocate open access versionFindings
  • Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, pp. 3105–3113 (2015)
    Google ScholarLocate open access versionFindings
  • Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In: Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1058–1066 (2013)
    Google ScholarLocate open access versionFindings
  • Baldassi, C., Ingrosso, A., Lucibello, C., Saglietti, L., Zecchina, R.: Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses. Phys. Rev. Lett. 115(12), 128101 (2015)
    Google ScholarLocate open access versionFindings
  • Kim, M., Smaragdis, P.: Bitwise neural networks. arXiv preprint arXiv:1601.06071 (2016)
    Findings
  • Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
    Findings
  • Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
    Findings
  • Redmon, J.: Darknet: open source neural networks in c (2013–2016). http://pjreddie.com/darknet/
    Findings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科