Limitations of the empirical Fisher approximation for natural gradient descent
NeurIPS, pp. 4158-4169, 2019.
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-ord...More
PPT (Upload PPT)