Amortized Nesterov's Momentum - A Robust Momentum and Its Application to Deep Learning.
UAI(2020)
摘要
This work proposes a novel momentum technique, the Amortized Nesterov’s Momentum, for stochastic convex optimization. The proposed method can be regarded as a smooth transition between Nesterov’s method and mirror descent. By tuning only a single parameter, users can trade Nesterov’s acceleration for robustness, that is, the variance control of the stochastic noise. Motivated by the recent success of using momentum in deep learning, we conducted extensive experiments to evaluate this new momentum in deep learning tasks. The results suggest that it can serve as a favorable alternative for Nesterov’s momentum.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络