On the Convergence, Implicit Bias, and Edge of Stability of Gradient Descent in Deep Learning: Reviewing Recent Progress [special Issue on the Mathematics of Deep Learning]

Deep neural networks (DNNs) trained via gradient descent (GD) with random initialization and without any regularization enjoy good generalization performance in practice despite being highly overparametrized. To theoretically understand this puzzling phenomenon, many works on convergence analysis for GD algorithms on NNs have been developed over the last half-decade. In this article, we review these research efforts and discuss how they address three specific questions related to this puzzle. The first question is why GD finds a global minimum efficiently, which the literature has addressed by studying what level of overparametrization (width, depth, etc.) and what type of initialization lead to a benign optimization landscape along the training trajectory, facilitating a linear convergence rate of GD. The next question is why the global minimum found by GD generalizes well, which has been addressed by showing that overparametrization induces an implicit simplicity bias along the GD trajectory. More recently, it has been observed that in practice, training DNNs with far larger learning rates than theoretically permissible results in faster convergence and better generalization. This leads to the third question of why faster convergence and better generalization can be achieved with a large learning rate, which has been recently addressed by identifying a self-stabilization mechanism and implicit bias toward flat minima.