Unveiling Hidden Convexity in Deep Learning: A Sparse Signal Processing Perspective [special Issue on the Mathematics of Deep Learning]

Deep neural networks (DNNs) have achieved groundbreaking success in various fields. Yet, the inherent nonconvexity of their training objectives and the complexity of their architectures present challenges for optimization and theoretical understanding. Our article examines the hidden convex structure within DNNs by drawing upon frameworks from sparse signal processing, specifically least absolute shrinkage and selection operator (LASSO), group LASSO, and nuclear-norm regularized models. By leveraging these established techniques from sparse linear models, we demonstrate how convex formulations can yield global optima and facilitate the interpretability of deep networks. Furthermore, we explore convolutional neural networks (CNNs), transformers, diffusion models, and generative adversarial networks from this convex regularization framework. We show how hypercomplex structures, such as geometric algebra, emerge in this analysis, offering new insights into the generalization to unseen data. We aim to engage the signal processing community in exploring these convex perspectives to better understand deep learning and its practical applications.