Visual Autoregressive (VAR) models have recently demonstrated impressive image generation quality while maintaining low latency. However, they suffer from severe KV-cache memory constraints, often requiring gigabytes of memory per generated image. We introduce HeatKV, a novel compression method that adapts cache allocation in each head based on its attention to previously generated scales. Using a small offline calibration set, the attention heads are ranked according to their attention scores over prior scales. Based on this ranking, we construct a static pruning schedule tailored to a given memory budget. Applied to the Infinity-2B model, HeatKV achieves 2 × higher compression ratio in memory allocation for KV cache compared to existing methods, while maintaining similar or better image fidelity, prompt alignment and human perception score. Our method achieves a new state-of-the-art (SOTA) for VAR model KV-cache compression, showcasing the effectiveness of fine-grained, head-specific cache allocation. Code and calibration script available at https://github.com/arm-research/heatkv.

We develop a Lyapunov-based analysis of Korpelevich's extragradient method and show that it achieves an o(1/k) last-iterate convergence rate of the constructed Lyapunov function. This Lyapunov function simultaneously upper bounds several standard measures of optimality, which allows our analysis to sharpen existing last-iterate convergence guarantees for these measures. Moreover, the same analysis enables the design of a class of flexible extensions of the extragradient method in which extragradient steps are adaptively blended with user-specified directions via a Lyapunov-guided line-search procedure. These extensions retain global convergence under practical assumptions and can attain superlinear rates when the directions are chosen appropriately. Numerical experiments confirm the simplicity and efficiency of the proposed framework.

We introduce AutoLyap, a software suite that assists with Lyapunov analyses of a wide class of first-order methods for structured optimization and inclusion problems. Lyapunov analyses are structured proof patterns, with historical roots in the study of dynamical systems, commonly used to establish convergence results for first-order methods. Building on previous work, the core idea behind AutoLyap is to recast the verification of the existence of a Lyapunov analysis as a semidefinite program (SDP), which can then be solved numerically using standard SDP solvers. Users of the package specify (i) the class of optimization or inclusion problems, (ii) the first-order method in question, and (iii) the type of Lyapunov analysis they wish to test. Once these inputs are provided, AutoLyap handles the SDP modeling and proceeds to solve the SDP numerically. We use the package to numerically verify and extend several convergence results. AutoLyap is currently available in Python and Julia.

We study frugal splitting algorithms with minimal lifting for solving monotone inclusion problems involving sums of maximal monotone and cocoercive operators. Building on a foundational result by Ryu, we fully characterize all methods that use only individual resolvent evaluations, direct evaluations of cocoercive operators, and minimal memory resources while ensuring convergence via averaged fixed-point iterations. We show that all such methods are captured by a unified framework, which includes known schemes and enables new ones with promising features. Systematic numerical experiments lead us to propose three design heuristics to achieve excellent performances in practice, yielding significant gains over existing methods.

Variance-reduced stochastic gradient methods have gained popularity in recent times. Several variants exist with different strategies for storing and sampling gradients, and this work concerns the interactions between these two aspects. We present a general proximal variance-reduced gradient method and analyze it under strong convexity assumptions. Special cases of the algorithm include SAGA, L-SVRG, and their proximal variants. Our analysis sheds light on epoch-length selection and the need to balance the convergence of the iterates with how often gradients are stored. The analysis improves on other convergence rates found in the literature and produces a new and faster converging sampling strategy for SAGA. Problem instances for which the predicted rates are the same as the practical rates are presented together with problems based on real-world data.

The Chambolle–Pock method is a versatile three-parameter algorithm designed to solve a broad class of composite convex optimization problems, which encompass two proper, lower semicontinuous, and convex functions, along with a linear operator L . The functions are accessed via their proximal operators, while the linear operator is evaluated in a forward manner. Among the three algorithm parameters $$\tau $$ , $$\sigma $$ , and $$\theta $$ ; $$\tau , \sigma >0$$ serve as step sizes for the proximal operators, and $$\theta $$ is an extrapolation step parameter. Previous convergence results have been based on the assumption that $$\theta =1$$ . We demonstrate that weak convergence is achievable whenever $$\theta > 1/2$$ and $$\tau \sigma \Vert L\Vert ^2<4/\mathord {\left( 1+2\theta \right) }$$ . Moreover, we establish tightness of the step size bound by providing an example that is nonconvergent whenever the second bound is violated.

The Douglas-Rachford splitting method is a classical and widely used algorithm for solving monotone inclusions involving the sum of two maximally monotone operators. It was recently shown to be the unique frugal, no-lifting resolvent-splitting method that is unconditionally convergent in the general two-operator setting. In this work, we show that this uniqueness does not hold in the convex optimization case: when the operators are subdifferentials of proper, closed, convex functions, a strictly larger class of frugal, no-lifting resolvent-splitting methods is unconditionally convergent. We provide a complete characterization of all such methods in the convex optimization setting and prove that this characterization is sharp: unconditional convergence holds exactly on the identified parameter regions. These results immediately yield new families of convergent ADMM-type and Chambolle-Pock-type methods obtained through their Douglas-Rachford reformulations.

The Chambolle–Pock method is a versatile three-parameter algorithm designed to solve a broad class of composite convex optimization problems, which encompass two proper, lower semicontinuous, and convex functions, along with a linear operator L. The functions are accessed via their proximal operators, while the linear operator is evaluated in a forward manner. Among the three algorithm parameters τ , σ , and θ ; τ , σ >0 serve as step sizes for the proximal operators, and θ is an extrapolation step parameter. Previous convergence results have been based on the assumption that θ =1 . We demonstrate that weak convergence is achievable whenever θ > 1/2 and τσ‖ L‖ ^2<4/( 1+2θ) . Moreover, we establish tightness of the step size bound by providing an example that is nonconvergent whenever the second bound is violated.

We propose and study a weakly convergent variant of the forward--backward algorithm for solving structured monotone inclusion problems. Our algorithm features a per-iteration deviation vector which provides additional degrees of freedom. The only requirement on the deviation vector to guarantee convergence is that its norm is bounded by a quantity that can be computed online. This approach provides great flexibility and opens up for the design of new and improved forward--backward-based algorithms, while retaining global convergence guarantees. These guarantees include linear convergence of our method under a metric subregularity assumption without the need to adapt the algorithm parameters. Choosing suitable monotone operators allows for incorporating deviations into other algorithms, such as Chambolle--Pock and Krasnoselsky--Mann iterations. We propose a novel inertial primal--dual algorithm by selecting the deviations along a momentum direction and deciding their size using the norm condition. Numerical experiments demonstrate our convergence claims and show that even this simple choice of deviation vector can improve the performance, compared, e.g., to the standard Chambolle--Pock algorithm.

We present a methodology for establishing the existence of quadratic Lyapunov inequalities for a wide range of first-order methods used to solve convex optimization problems. In particular, we consider (i) classes of optimization problems of finite-sum form with (possibly strongly) convex and possibly smooth functional components, (ii) first-order methods that can be written as a linear system on state-space form in feedback interconnection with the subdifferentials of the functional components of the objective function, and (iii) quadratic Lyapunov inequalities that can be used to draw convergence conclusions. We present a necessary and sufficient condition for the existence of a quadratic Lyapunov inequality within a predefined class of Lyapunov inequalities, which amounts to solving a small-sized semidefinite program. We showcase our methodology on several first-order methods that fit the framework. Most notably, our methodology allows us to significantly extend the region of parameter choices that allow for duality gap convergence in the Chambolle–Pock method when the linear operator is the identity mapping.

We investigate frugal splitting operators for finite sum monotone inclusion problems. These operators utilize exactly one direct or resolvent evaluation of each operator of the sum, and the splitting operator's output is dictated by linear combinations of these evaluations' inputs and outputs. To facilitate analysis, we introduce a novel representation of frugal splitting operators via a generalized primal-dual resolvent. The representation is characterized by an index and four matrices, and we provide conditions on these that ensure equivalence between the classes of frugal splitting operators and generalized primal-dual resolvents. Our representation paves the way for new results regarding lifting numbers and the development of a unified convergence analysis for frugal splitting operator methods, contingent on the directly evaluated operators being cocoercive. The minimal lifting number is n - 1 - f where n is the number of monotone operators and f is the number of direct evaluations in the splitting. Notably, this lifting number is achievable only if the first and last operator evaluations are resolvent evaluations. These results generalize the minimal lifting results by Ryu and by Malitsky and Tam that consider frugal resolvent splittings. Building on our representation, we delineate a constructive method to design frugal splitting operators, exemplified in the design of a novel, convergent, and parallelizable frugal splitting operator with minimal lifting.

In this paper, we extend the previous convergence results for the generalized alternating projection method applied to subspaces in [arXiv:1703.10547] to hold also for smooth manifolds. We show that the algorithm locally behaves similarly in the subspace and manifold settings and that the same rates are obtained. We also present convergence rate results for when the algorithm is applied to non-empty, closed, and convex sets. The results are based on a finite identification property that implies that the algorithm after an initial identification phase solves a smooth manifold feasibility problem. Therefore, the rates in this paper hold asymptotically for problems in which this identification property is satisfied. We present a few examples where this is the case and also a counter example for when this is not.

Progenitor cells adapt their behavior in response to tissue demands. However, the molecular mechanisms controlling esophageal progenitor decisions remain largely unknown. Here, we demonstrate the presence of a Troy (Tnfrsf19)-expressing progenitor subpopulation localized to defined regions along the mouse esophageal axis. Lineage tracing and mathematical modeling demonstrate that Troy-positive progenitor cells are prone to undergoing symmetrical fate choices and contribute to esophageal tissue homeostasis long term. Functionally, TROY inhibits progenitor proliferation and enables commitment to differentiation without affecting fate symmetry. Whereas Troy expression is stable during esophageal homeostasis, progenitor cells downregulate Troy in response to tissue stress, enabling proliferative expansion of basal cells refractory to differentiation and reestablishment of tissue homeostasis. Our results demonstrate functional, spatially restricted progenitor heterogeneity in the esophageal epithelium and identify how dynamic regulation of Troy coordinates tissue generation.

It is shown that the behavior of an m-port circuit of maximal monotone elements can be expressed as a zero of the sum of a maximal monotone operator containing the circuit elements, and a structured skew-symmetric linear operator representing the interconnection structure, together with a linear output transformation. The Condat–Vũ algorithm solves inclusion problems of this form, and may be used to solve for the periodic steady-state behavior, given a periodic excitation at each port, using an iteration in the space of periodic trajectories.

We propose a variation of the forward–backward splitting method for solving structured monotone inclusions. Our method integrates past iterates and two deviation vectors into the update equations. These deviation vectors bring flexibility to the algorithm and can be chosen arbitrarily as long as they together satisfy a norm condition. We present special cases where the deviation vectors, selected as predetermined linear combinations of previous iterates, always meet the norm condition. Notably, we introduce an algorithm employing a scalar parameter to interpolate between the conventional forward–backward splitting scheme and an accelerated 𝒪( 1/n^2) -convergent forward–backward method that encompasses both the accelerated proximal point method and the Halpern iteration as special cases. The existing methods correspond to the two extremes of the allowed scalar parameter range. By choosing the interpolation scalar near the midpoint of the permissible range, our algorithm significantly outperforms these previously known methods when addressing a basic monotone inclusion problem stemming from minimax optimization.

The nonlinear, or warped, resolvent recently explored by Giselsson and Bùi-Combettes has been used to model a large set of existing and new monotone inclusion algorithms. To establish convergent algorithms based on these resolvents, corrective projection steps are utilized in both works. We present a different way of ensuring convergence by means of a nonlinear momentum term, which in many cases leads to cheaper per-iteration cost. The expressiveness of our method is demonstrated by deriving a wide range of special cases. These cases cover and expand on the forward-reflected-backward method of Malitsky-Tam, the primal-dual methods of Vũ-Condat and Chambolle-Pock, and the forward-reflected-Douglas-Rachford method of Ryu-Vũ. A new primal-dual method that uses an extra resolvent step is also presented as well as a general approach for adding momentum to any special case of our nonlinear forward-backward method, in particular all the algorithms listed above.

Many mathematical formulations of significant real-world technological or physical problems are based on optimization with objective functions being the sum of several functions, each with different properties.This structure allows to employ splitting methods in which the problem is split into parts that can be treated more efficiently than the full problem via gradients, projections, proximal evaluations, and resolvents.Different splitting methods-such as the alternating direction method of multipliers (ADMM), the Douglas-Rachford method, and forward-backward splitting-iteratively combine such evaluations in different ways to solve the original problem.This special issue brings together ten articles on recent advances in projection and splitting methods.The results present new approaches to challenging mathematical problems and applications in various fields such as variational inequalities, fixed point theory, and inverse problems.A summary of these articles in alphabetical order by first author is presented next.H.H. Bauschke, H. Ouyang and X. Wang, in On angles between convex cones, present the extensions of results relating the Dixmier and Friedrichs angles associated with pairs of linear subspaces to the special case where those subspaces are convex cones.These angles are an essential tool in analyzing the convergence rates of projection-based algorithms.S. Deyo and V. Elser, in Avoiding traps in nonconvex problems, provide the guidelines on the tuning of parameters for iterative projection algorithms when they are applied to nonconvex problems.The importance of tuning both hyperparameters and metric parameters is discussed, and illustrated through examples.A. Gibali and Y.I.Suleiman, in Parallel projection method for solving split equilibrium problems, introduce a new class of bilevel multiple sets split equilibrium problems and present a parallel projection method for solving them.A proof of convergence and numerical illustration is also provided.B. Peters, in Point-to-set distance functions for output-constrained neural networks, develops a new algorithm to constrain the output of neural networks for semantic segmentation based on prior information.The method achieves this through the use of projection-based point-to-set distance functions.The effectiveness of the method is highlighted with relevant examples from image processing and geosciences.

We propose a novel dynamically weighted inertial forward-backward algorithm (DWIFOB) for solving structured monotone inclusion problems. The scheme exploits the globally convergent forward-backward algorithm with deviations in [26] as the basis and combines it with the extrapolation technique used in Anderson acceleration to improve local convergence. We also present a globally convergent primal-dual variant of DWIFOB and numerically compare its performance to the primal-dual method of Chambolle-Pock and a Tikhonov regularized version of Anderson acceleration applied to the same problem. In all our numerical evaluations, the primal-dual variant of DWIFOB outperforms the Chambolle-Pock algorithm. Moreover, our numerical experiments suggest that our proposed method is much more robust than the regularized Anderson acceleration, which can fail to converge and be sensitive to algorithm parameters. These numerical experiments highlight that our method performs very well while still being robust and reliable.

With the purpose of examining biased updates in variance-reduced stochastic gradient methods, we introduce SVAG, a SAG/SAGA-like method with adjustable bias. SVAG is analyzed in a cocoercive root-finding setting, a setting which yields the same results as in the usual smooth convex optimization setting for the ordinary proximal-gradient method. We show that the same is not true for SVAG when biased updates are used. The step-size requirements for when the operators are gradients are significantly less restrictive compared to when they are not. This highlights the need to not rely solely on cocoercivity when analyzing variance-reduced methods meant for optimization. Our analysis either match or improve on previously known convergence conditions for SAG and SAGA. However, in the biased cases they still do not correspond well with practical experiences and we therefore examine the effect of bias numerically on a set of classification problems. The choice of bias seem to primarily affect the early stages of convergence and in most cases the differences vanish in the later stages of convergence. However, the effect of the bias choice is still significant in a couple of cases.

In this paper, we present and compare three novel model-cum-data-driven channel estimation procedures in a millimeter-wave Multi-Input Multi-Output (MIMO) Orthogonal Frequency Division Multiplexing (OFDM) wireless communication system. The transceivers employ a hybrid analog-digital architecture. We adapt techniques from a wide range of signal processing methods, such as detection and estimation theories, compressed sensing, and Bayesian inference, to learn the unknown virtual beamspace domain dictionary, as well as the delay-and-beamspace sparse channel. We train the model-based algorithms with a site-specific training dataset generated using a realistic ray tracing-based wireless channel simulation tool. We assess the performance of the proposed channel estimation algorithms with the same site's test data. We benchmark the performance of our novel procedures in terms of normalized mean squared error against an existing fast greedy method and empirically show that model-based approaches combined with data-driven customization unanimously outperform the state-of-the-art techniques by a large margin. The proposed algorithms were selected as the top three solutions in the "ML5G-PHY Channel Estimation Global Challenge 2020" organized by the International Telecommunication Union.