Entropy-SGD: biasing gradient descent into wide valleys

Journal of Statistical Mechanics: Theory and Experiment, pp. 1240182019.

Cited by: 281|Bibtex|Views138|DOI:https://doi.org/10.1088/1742-5468/ab39d9
Other Links: academic.microsoft.com|arxiv.org

Abstract:

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage u...More

Code:

Data:

Your rating :
0

 

Tags
Comments