Infer2Train: leveraging inference for better training of deep networks

Training large scale Deep Neural Networks (DNNs) requires ever growing computational resources. This growth is usually based on larger and faster training devices. However, a new category of inference-only accelerators is emerging, allowing fast and energy efficient forward pass using low precision operations. In this study, we explore how to leverage such inference-only accelerators for improving training performance. We examine several alternatives and show preliminary results with improved test accuracy on visual-classification tasks such as training ResNet model on the ImageNet and Cifar datasets.