Approximate message passing (AMP) provides fast iterative algorithms for high-dimensional signal recovery with Gaussian design matrices, while the Convex Gaussian Min-max Theorem (CGMT) gives a static optimization framework for obtaining sharp asymptotic characterizations of convex estimators. Although these two frameworks often lead to the same scalar state-evolution equations, their connection is usually indirect. In this paper, we establish a direct connection between the two for regularized linear regression in the proportional high-dimensional regime. When the CGMT Auxiliary Optimization (AO) and Primary Optimization (PO) give the same primal-dual solution, we show that the CGMT framework recovers the AMP fixed-point equations, including the Onsager correction. We further identify the AO Gaussian vectors with the Gaussian perturbations in the primal and residual AMP channels. For regularized M-estimation, the same viewpoint recovers the fixed point of scalar-variance max-sum Generalized AMP (GAMP). These results show that the AMP (and GAMP) iterations are suggested, and can be derived, from the CGMT framework, and may further suggest a way to derive AMP-like algorithms in settings where CGMT applies but standard AMP derivations are unavailable.

We investigate Gaussian Universality for data distributions generated via diffusion models. By Gaussian Universality we mean that the test error of a generalized linear model f(W) trained for a classification task on the diffusion data matches the test error of f(W) trained on the Gaussian Mixture with matching means and covariances per class. In other words, the test error depends only on the first and second order statistics of the diffusion-generated data in the linear setting. As a corollary, the analysis of the test error for linear classifiers can be reduced to Gaussian data from diffusion-generated data. Analyzing the performance of models trained on synthetic data is a pertinent problem due to the surge of methods such as [Sehwag, et al. (2024)]. Moreover, we show that, for any 1- Lipschitz scalar function phi, phi(x) is close to E phi(x) with high probability for x sampled from the conditional diffusion model corresponding to each class. Finally, we note that current approaches for proving universality do not apply to diffusion-generated data as the covariance matrices of the data tend to have vanishing minimum singular values, contrary to the assumption made in the literature. This leaves extending previous mathematical universality results as an intriguing open question.

In the present paper we study the performance of linear denoisers for noisy data of the form 𝐱 + 𝐳, where 𝐱∈ℝ^d is the desired data with zero mean and unknown covariance , and 𝐳∼𝒩(0, _𝐳) is additive noise. Since the covariance is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples 𝐱_1,…,𝐱_n ∈ℝ^d from the true distribution. A standard approach would then be to estimate from the samples and use it to construct an “empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser 𝐖 from the data itself. In particular, we synthetically construct noisy samples 𝐱̂_i of the data by injecting the samples with Gaussian noise with covariance _1 ≠_𝐳 and find the best 𝐖 that approximates 𝐖𝐱̂_i ≈𝐱_i in a least-squares sense. In the proportional regime n/d→ κ> 1 we use the Convex Gaussian Min-Max Theorem (CGMT) to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over _1 to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the “empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as κ→∞.

K-means clustering is a workhorse of unsupervised learning, but it is notoriously brittle to outliers, distribution shifts, and limited sample sizes. Viewing k-means as Lloyd–Max quantization of the empirical distribution, we develop a distributionally robust variant that protects against such pathologies. We posit that the unknown population distribution lies within a Wasserstein-2 ball around the empirical distribution. In this setting, one seeks cluster centers that minimize the worst-case expected squared distance over this ambiguity set, leading to a minimax formulation. A tractable dual yields a soft-clustering scheme that replaces hard assignments with smoothly weighted ones. We propose an efficient block coordinate descent algorithm with provable monotonic decrease and local linear convergence. Experiments on standard benchmarks and large-scale synthetic data demonstrate substantial gains in outlier detection and robustness to noise.

We study online linear-quadratic regulation (LQR) with unknown dynamics under communication rate constraints. Classical networked control quantizes the plant state at every time step, requiring O(T) total bits while injecting persistent quantization noise that limits control performance. We consider a setting where the plant observes its state locally and can estimate system dynamics via ordinary least squares, while a remote controller possesses knowledge of the control cost. Rather than quantizing the raw state, the plant transmits learned dynamics estimates over a rate-limited uplink, and the controller returns the optimal control policy so that the plant can compute actions locally using its superior state knowledge. We first prove a fundamental information-theoretic lower bound: any scheme achieving O(T^α) regret for α∈ [1/2,1) compared to the optimal infinite horizon LQR controller that knows the true system dynamics must transmit at least Ω(log T) bits. We then design the Quantized Certainty Equivalent (QCE-LQR) algorithm, which matches this bound. The resulting regret bound contains inflation factors Q_slow(ϱ) and Q_fast(ϱ) that vanish as the codebook resolution increases, smoothly recovering the unquantized baseline regret. Numerical experiments on four benchmark systems – from a scalar unstable plant to a 24-parameter Boeing 747 lateral model – confirm that a variant of QCE-LQR achieves regret comparable to an unquantized certainty equivalent controller over a horizon of T=10,000 steps.

We study linear codes that maximize minimum distance subject to arbitrary support constraints on the parity-check matrix. Such constraints arise naturally in the design of LDPC codes, locally repairable codes, and hardware-constrained systems where each parity check must involve only a limited number of code symbols. They are also essential in quantum error correction, where sparse stabilizers reduce measurement noise and respect the connectivity constraints of physical qubit architectures. We derive the optimal minimum distance possible given support constraints on the parity-check matrix and show it is achievable over sufficiently large fields. When this maximum distance coincides with the Singleton bound for unconstrained parity check matrices, the dual GM-MDS construction yields generalized Reed–Solomon codes obeying the mask. In the generator-matrix setting, the GM-MDS theorem guarantees that the optimal distance can always be achieved by a subcode of a generalized Reed–Solomon code while satisfying arbitrary support constraints. We show that this is not true for the parity-check setting. We exhibit a set of support constraints, derived from the vertex-edge incidence of K_6,6, for which the optimal minimum distance cannot be realized by any subcode of a generalized Reed–Solomon code over any field. We also analyze structured constraint families – regular, balanced, and cyclic masks – through numerical optimization, providing design guidance for practical code constructions.

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

In learning problems, the population loss is often used as a proxy for the true expected loss. However, in the high-dimensional regime (with a fixed data-to-dimension ratio), this proxy can be misleading due to the paucity or contamination of data. An alternative is distributionally robust training, which replaces empirical risk minimization with the minimization of a worst-case expected loss over all distributions in an ambiguity set around the empirical distribution. Wasserstein-based constructions of ambiguity sets and practical algorithms have been studied in [1]. In this work, we provide an asymptotic characterization of the statistical behavior of such distributionally robust procedures. We consider two settings. First, we study distributionally robust binary classification with a linear decision rule, where the classifier is learned by minimizing the worst-case expected convex margin loss over all distributions in a Wasserstein ball around the empirical distribution. Second, we study distributionally robust estimation for binary generalized linear models, where labels are generated from an underlying ground-truth linear signal through a binary link function. The goal is to recover this latent signal from data. Across both settings, our main contribution is an asymptotically exact characterization that reduces the high-dimensional classification and estimation errors to a scalar optimization problem. This scalar formulation applies to a broad class of convex losses, including logistic loss and modified Huber loss, and makes explicit how Wasserstein robustness trades off with over-/under-parameterization, model mismatch, and corruption fraction c < 0.5 of the training labels.

In this work we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent, Gradient Clipping and Adam. We consider preconditioners of the form ∇ K, where K: ℝ^p →ℝ is convex and assume that the latter is applied to train an over-parameterized linear model with loss of the form ℓ(XW - Y), for weights W∈ℝ^d × k, labels Y∈ℝ^n × k and data X∈ℝ^n × d. Under the aforementioned assumptions, we prove that the iterates of the preconditioned gradient descent always converge to a point W_∞∈ℝ^d × k satisfying XW_∞ = Y. Our proof techniques are of independent interest as we introduce a novel version of the Bregman Divergence with accompanying identities that allow us to establish convergence. We also study the implicit bias of Dual Space Preconditioned Gradient Descent. First, we demonstrate empirically that, for general K(·), W_∞ depends on the chosen learning rate, hindering a precise characterization of the implicit bias. Then, for preconditioners of the form K(G) = h(G_F), known as isotropic preconditioners, we show that W_∞ minimizes W_∞ - W_0_F^2 subject to XW_∞ = Y, where W_0 is the initialization. Denoting the convergence point of GD initialized at W_0 by W_GD, ∞, we thus note W_∞ = W_GD, ∞ for isotropic preconditioners. Finally, we show that a similar fact holds for general preconditioners up to a multiplicative constant, namely, W_0 - W_∞_F ≤ c W_0 - W_GD, ∞_F for a constant c>0.

In this paper, we introduce a novel convex relaxation for the recovery of M-PAM signals in MIMO communications. The convex relaxation is inspired by the implicit bias property of the mirror descent algorithm for under-determined systems of linear equations. When M = 2, the relaxation reduces to the standard box relaxation commonly used in MIMO signal recovery. For M > 2, the new relaxation yields a recovery threshold that is strictly better than that of the box relaxation (especially when the probability of symbols is asymmetrical). For 4-PAM signals, under a Gaussianity assumption on the channel matrix, we provide a precise analysis of the recovery threshold and symbol-error-rate (SER) of the new method. Simulations confer the improvement in SER over the standard box relaxation when the symbol probabilities exhibit asymmetry.

We investigate Stochastic Mirror Descent (SMD) with matrix parameters and vector-valued predictions, a framework relevant to multi-class classification and matrix completion problems. Focusing on the overparameterized regime, where the total number of parameters exceeds the number of training samples, we prove that SMD with matrix mirror functions ψ(·) converges exponentially to a global interpolator. Furthermore, we generalize classical implicit bias results of vector SMD by demonstrating that the matrix SMD algorithm converges to the unique solution minimizing the Bregman divergence induced by ψ(·) from initialization subject to interpolating the data. These findings reveal how matrix mirror maps dictate inductive bias in high-dimensional, multi-output problems.

In data-driven learning and inference tasks, the high cost of acquiring samples from the target distribution often limits performance. A common strategy to mitigate this challenge is to augment the limited target samples with data from a more accessible "auxiliary" distribution. This paper establishes fundamental limits of this approach by analyzing the improvement in the mean square error (MSE) when estimating the mean of the target distribution. Using the Wasserstein-2 metric to quantify the distance between distributions, we derive expressions for the worst-case MSE when samples are drawn (with labels) from both a target distribution and an auxiliary distribution within a specified Wasserstein-2 distance from the target distribution. We explicitly characterize the achievable MSE and the optimal estimator in terms of the problem dimension, the number of samples from the target and auxiliary distributions, the Wasserstein-2 distance, and the covariance of the target distribution. We note that utilizing samples from the auxiliary distribution effectively improves the MSE when the squared radius of the Wasserstein-2 uncertainty ball is small compared to the variance of the true distribution and the number of samples from the true distribution is limited. Numerical simulations in the Gaussian location model illustrate the theoretical findings.

We study the use of linear regression for multiclass classification in the over-parametrized regime where some of the training data is mislabeled. In such scenarios it is necessary to add an explicit regularization term, lambda f (& sdot;), for some convex function f(& sdot;), to avoid overfitting the mislabeled data. In our analysis, we assume that the data is sampled from a Gaussian Mixture Model with equal class sizes, and that a proportion of the training labels is corrupted for each class. Under these assumptions, we prove that the best classification performance is achieved when f(& sdot;) = parallel to & sdot; parallel to(2)(2) and lambda -> infinity. We then proceed to analyze the classification errors for f(& sdot;) = parallel to & sdot; parallel to(1) and f(& sdot;) = parallel to & sdot; parallel to(infinity) in the large lambda regime and notice that it is often possible to find sparse and one-bit solutions, respectively, that perform almost as well as the one corresponding to f(& sdot;) = parallel to & sdot; parallel to(2)(2).

Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias of the optimization algorithm. The question we address in this paper is, “What is the implicit bias that leads to the best generalization performance?". To find the optimal implicit bias, we provide a precise asymptotic analysis of the generalization performance of interpolators obtained from the minimization of convex functions/potentials for over-parameterized linear regression with non-isotropic Gaussian data. In particular, we obtain a tight lower bound on the best generalization error possible among this class of interpolators in terms of the over-parameterization ratio, the variance of the noise in the labels, the eigenspectrum of the data covariance, and the underlying distribution of the parameter to be estimated. Finally, we find the optimal convex implicit bias that achieves this lower bound under certain sufficient conditions involving the log-concavity of the distribution of a Gaussian convolved with the prior of the true underlying parameter.

In the past couple of decades, the use of “non-quadratic" convex cost functions has revolutionized signal processing, machine learning, and statistics, allowing one to customize solutions to have desired structures and properties. However, the situation is not the same in control where the use of quadratic costs still dominates, ostensibly because determining the “value function", i.e., the optimal expected cost-to-go, which is critical to the construction of the optimal controller, becomes computationally intractable as soon as one considers general convex costs. As a result, practitioners often resort to heuristics and approximations, such as model predictive control that only looks a few steps into the future. In the quadratic case, the value function is easily determined by solving Riccati equations. In this work, we consider a special class of convex cost functions constructed from Bregman divergence and show how, with appropriate choices, they can be used to fully extend the framework developed for the quadratic case. The resulting optimal controllers are infinite horizon, come with stability guarantees, and have state-feedback, or estimated state-feedback, laws. They exhibit a much wider range of behavior than their quadratic counterparts since the feedback laws are nonlinear. The approach can be applied to several cases of interest, including safety control, sparse control, and bang-bang control.

Stochastic Mirror Descent (SMD) has emerged as a method for solving various convex optimization problems and, under appropriate conditions, has been shown to exponentially converge to the global optimum when the underlying loss is strongly convex. However, strong convexity of the loss fails to hold when the optimization problem is over a set of over-parameterized linear equations, a scenario that frequently occurs in modern machine learning and signal processing. In this setting, prior literature only guarantees much slower convergence rates. In this paper, we show exponential convergence of the SMD iterates to a global optimum (the one that exhibits the appropriate implicit bias) in such over-parameterized linear models. As an extension of this result, we show the exponential convergence of the related Regularizer Mirror Descent (RMD) iterates to the global optimum of explicitly regularized over-parameterized linear models.

This paper investigates distributionally robust filtering for state-space models subject to exogenous disturbances in state evolution and observation processes. The joint probability distribution of the disturbance process over an arbitrary horizon is unknown but is assumed to reside within a Wasserstein-2 ambiguity set centered around a nominal distribution. We seek to develop a causal estimator that minimizes the worst-case mean squared error (MSE) among all distributions in this ambiguity set. Unlike prior works, our framework accommodates disturbances with arbitrary temporal correlations. In the finite-horizon setting, this problem reduces to a semi-definite program (SDP) whose complexity scales with the time horizon. Consequently, we focus on the infinite-horizon case, where we derive the optimal linear time-invariant (LTI) filter using the Karush-Kuhn-Tucker (KKT) conditions and propose an efficient frequency-domain algorithm to compute it. While the optimal LTI filter generally has a non-rational transfer function, we provide a minimax rational approximation method to approximate the optimal non-rational filter, in the H8-norm, with a near-optimal finite-order state-space estimator. This approach avoids the computational challenges associated with horizon-dependent scaling in the finite-horizon case. Numerical simulations demonstrate the effectiveness of the proposed method.

A celebrated result by Gordon allows one to compare the min-max behavior of two Gaussian processes if certain inequality conditions are met. The consequences of this result include the Gaussian min-max (GMT) and convex Gaussian min-max (CGMT) theorems which have had far-reaching implications in high-dimensional statistics, machine learning, non-smooth optimization, and signal processing. Both theorems rely on a pair of Gaussian processes, first identified by Slepian, that satisfy Gordon's comparison inequalities. In this paper, we identify a new pair of Gaussian processes satisfying these inequalities. The resulting theorems extend the classical GMT and CGMT Theorems from the case where the underlying Gaussian matrix in the primary process has iid rows to where it has independent but non-identically-distributed ones. The new CGMT is applied to the problems of multi-source Gaussian regression, as well as to binary classification of general Gaussian mixture models.

In the past couple of decades, non-quadratic convex penalties have reshaped signal processing and machine learning; in robust control, however, general convex costs break the Riccati and value function structure that make the design tractable. Practitioners thus default to approximations, heuristics or robust model predictive control that are solved online for short horizons. We close this gap by extending H-infinity control of discrete-time linear systems to strictly convex penalties on state, input, and disturbance, recasting the objective with Bregman divergences that admit a completion-of-squares decomposition. The result is a closed-form, time-invariant, full-information stabilizing controller that minimizes a worst-case performance ratio over the infinite horizon. Necessary and sufficient existence/optimality conditions are given by a Riccati-like identity together with a concavity requirement; with quadratic costs, these collapse to the classical H-infinity algebraic Riccati equation and the associated negative-semidefinite condition, recovering the linear central controller. Otherwise, the optimal controller is nonlinear and can enable safety envelopes, sparse actuation, and bang-bang policies with rigorous H-infinity guarantees.