Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), 2019.
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-ord...More
Get fulltext within 24h
Full Text (Upload PDF)
PPT (Upload PPT)