A Computationally Efficient Second Order Deep Learning Training Algorithm with Adaptive Momentum

In this paper we evaluate the performance of an efficient second order algorithm for training deep neural networks. Unlike traditional SGD with momentum, the algorithm incorporates two significant modifications originating from treating the training task as a constrained optimization problem: Firstly, it scales the momentum term with curvature information, derived from the Hessian. Secondly, it dynamically adjusts coefficients for the learning rate and the scaled momentum term. The implementation of the algorithm requires minimal additional computations compared to classical SGD with momentum iterations since it only involves computing a Hessian-vector product rather than the entire Hessian matrix, which can be efficiently computed within any modern computational graph framework. We present experimental results on various neural network architectures trained on standard benchmarks which demonstrate the effectiveness of the proposed approach.