Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), pp. 8080-8091, 2019.
Natural gradient descent has proven effective at mitigating the effects of pathological curvature in neural network optimization, but little is known theoretically about its convergence properties, especially for nonlinear networks. In this work, we analyze for the first time the speed of convergence of natural gradient descent on nonline...More
PPT (Upload PPT)