On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights
arxiv(2023)
摘要
We revisit the general framework introduced by Fazylab et al. (SIAM J. Optim.
28, 2018) to construct Lyapunov functions for optimization algorithms in
discrete and continuous time. For smooth, strongly convex objective functions,
we relax the requirements necessary for such a construction. As a result we are
able to prove for Polyak's ordinary differential equations and for a
two-parameter family of Nesterov algorithms rates of convergence that improve
on those available in the literature. We analyse the interpretation of Nesterov
algorithms as discretizations of the Polyak equation. We show that the
algorithms are instances of Additive Runge-Kutta integrators and discuss the
reasons why most discretizations of the differential equation do not result in
optimization algorithms with acceleration. We also introduce a modification of
Polyak's equation and study its convergence properties. Finally we extend the
general framework to the stochastic scenario and consider an application to
random algorithms with acceleration for overparameterized models; again we are
able to prove convergence rates that improve on those in the literature.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要