The Chambolle–Pock method is a versatile three-parameter algorithm designed to solve a broad class of composite convex optimization problems, which encompass two proper, lower semicontinuous, and convex functions, along with a linear operator L. The functions are accessed via their proximal operators, while the linear operator is evaluated in a forward manner. Among the three algorithm parameters τ , σ , and θ ; τ , σ >0 serve as step sizes for the proximal operators, and θ is an extrapolation step parameter. Previous convergence results have been based on the assumption that θ =1 . We demonstrate that weak convergence is achievable whenever θ > 1/2 and τσ‖ L‖ ^2<4/( 1+2θ) . Moreover, we establish tightness of the step size bound by providing an example that is nonconvergent whenever the second bound is violated.

We propose and study a weakly convergent variant of the forward--backward algorithm for solving structured monotone inclusion problems. Our algorithm features a per-iteration deviation vector which provides additional degrees of freedom. The only requirement on the deviation vector to guarantee convergence is that its norm is bounded by a quantity that can be computed online. This approach provides great flexibility and opens up for the design of new and improved forward--backward-based algorithms, while retaining global convergence guarantees. These guarantees include linear convergence of our method under a metric subregularity assumption without the need to adapt the algorithm parameters. Choosing suitable monotone operators allows for incorporating deviations into other algorithms, such as Chambolle--Pock and Krasnoselsky--Mann iterations. We propose a novel inertial primal--dual algorithm by selecting the deviations along a momentum direction and deciding their size using the norm condition. Numerical experiments demonstrate our convergence claims and show that even this simple choice of deviation vector can improve the performance, compared, e.g., to the standard Chambolle--Pock algorithm.

We present a methodology for establishing the existence of quadratic Lyapunov inequalities for a wide range of first-order methods used to solve convex optimization problems. In particular, we consider (i) classes of optimization problems of finite-sum form with (possibly strongly) convex and possibly smooth functional components, (ii) first-order methods that can be written as a linear system on state-space form in feedback interconnection with the subdifferentials of the functional components of the objective function, and (iii) quadratic Lyapunov inequalities that can be used to draw convergence conclusions. We present a necessary and sufficient condition for the existence of a quadratic Lyapunov inequality within a predefined class of Lyapunov inequalities, which amounts to solving a small-sized semidefinite program. We showcase our methodology on several first-order methods that fit the framework. Most notably, our methodology allows us to significantly extend the region of parameter choices that allow for duality gap convergence in the Chambolle–Pock method when the linear operator is the identity mapping.

We investigate frugal splitting operators for finite sum monotone inclusion problems. These operators utilize exactly one direct or resolvent evaluation of each operator of the sum, and the splitting operator's output is dictated by linear combinations of these evaluations' inputs and outputs. To facilitate analysis, we introduce a novel representation of frugal splitting operators via a generalized primal-dual resolvent. The representation is characterized by an index and four matrices, and we provide conditions on these that ensure equivalence between the classes of frugal splitting operators and generalized primal-dual resolvents. Our representation paves the way for new results regarding lifting numbers and the development of a unified convergence analysis for frugal splitting operator methods, contingent on the directly evaluated operators being cocoercive. The minimal lifting number is n - 1 - f where n is the number of monotone operators and f is the number of direct evaluations in the splitting. Notably, this lifting number is achievable only if the first and last operator evaluations are resolvent evaluations. These results generalize the minimal lifting results by Ryu and by Malitsky and Tam that consider frugal resolvent splittings. Building on our representation, we delineate a constructive method to design frugal splitting operators, exemplified in the design of a novel, convergent, and parallelizable frugal splitting operator with minimal lifting.

We propose several deep-learning accelerated optimization solvers with convergence guarantees. We use ideas from the analysis of accelerated forward-backward schemes like FISTA, but instead of the classical approach of proving convergence for a choice of parameters, such as a step-size, we show convergence whenever the update is chosen in a specific set. Rather than picking a point in this set using some predefined method, we train a deep neural network to pick the best update. Finally, we show that the method is applicable to several cases of smooth and non-smooth optimization and show superior results to established accelerated solvers.

It is shown that the behavior of an m-port circuit of maximal monotone elements can be expressed as a zero of the sum of a maximal monotone operator containing the circuit elements, and a structured skew-symmetric linear operator representing the interconnection structure, together with a linear output transformation. The Condat–Vũ algorithm solves inclusion problems of this form, and may be used to solve for the periodic steady-state behavior, given a periodic excitation at each port, using an iteration in the space of periodic trajectories.

We propose a variation of the forward–backward splitting method for solving structured monotone inclusions. Our method integrates past iterates and two deviation vectors into the update equations. These deviation vectors bring flexibility to the algorithm and can be chosen arbitrarily as long as they together satisfy a norm condition. We present special cases where the deviation vectors, selected as predetermined linear combinations of previous iterates, always meet the norm condition. Notably, we introduce an algorithm employing a scalar parameter to interpolate between the conventional forward–backward splitting scheme and an accelerated 𝒪( 1/n^2) -convergent forward–backward method that encompasses both the accelerated proximal point method and the Halpern iteration as special cases. The existing methods correspond to the two extremes of the allowed scalar parameter range. By choosing the interpolation scalar near the midpoint of the permissible range, our algorithm significantly outperforms these previously known methods when addressing a basic monotone inclusion problem stemming from minimax optimization.

The nonlinear, or warped, resolvent recently explored by Giselsson and Bùi-Combettes has been used to model a large set of existing and new monotone inclusion algorithms. To establish convergent algorithms based on these resolvents, corrective projection steps are utilized in both works. We present a different way of ensuring convergence by means of a nonlinear momentum term, which in many cases leads to cheaper per-iteration cost. The expressiveness of our method is demonstrated by deriving a wide range of special cases. These cases cover and expand on the forward-reflected-backward method of Malitsky-Tam, the primal-dual methods of Vũ-Condat and Chambolle-Pock, and the forward-reflected-Douglas-Rachford method of Ryu-Vũ. A new primal-dual method that uses an extra resolvent step is also presented as well as a general approach for adding momentum to any special case of our nonlinear forward-backward method, in particular all the algorithms listed above.

We propose a novel dynamically weighted inertial forward-backward algorithm (DWIFOB) for solving structured monotone inclusion problems. The scheme exploits the globally convergent forward-backward algorithm with deviations in [26] as the basis and combines it with the extrapolation technique used in Anderson acceleration to improve local convergence. We also present a globally convergent primal-dual variant of DWIFOB and numerically compare its performance to the primal-dual method of Chambolle-Pock and a Tikhonov regularized version of Anderson acceleration applied to the same problem. In all our numerical evaluations, the primal-dual variant of DWIFOB outperforms the Chambolle-Pock algorithm. Moreover, our numerical experiments suggest that our proposed method is much more robust than the regularized Anderson acceleration, which can fail to converge and be sensitive to algorithm parameters. These numerical experiments highlight that our method performs very well while still being robust and reliable.

In this work, we consider methods for solving large-scale optimization problems with a possibly nonsmooth objective function. The key idea is to first specify a class of optimization algorithms using a generic iterative scheme involving only linear operations and applications of proximal operators. This scheme contains many modern primal-dual first-order solvers like the Douglas-Rachford and hybrid gradient methods as special cases. Moreover, we show convergence to an optimal point for a new method which also belongs to this class. Next, we interpret the generic scheme as a neural network and use unsupervised training to learn the best set of parameters for a specific class of objective functions while imposing a fixed number of iterations. In contrast to other approaches of "learning to optimize", we present an approach which learns parameters only in the set of convergent schemes. As use cases, we consider optimization problems arising in tomographic reconstruction and image deconvolution, and in particular a family of total variation regularization problems.

We investigate the techniques and ideas used in Shefi and Teboulle (SIAM J Optim 24(1), 269–297, 2014) in the convergence analysis of two proximal ADMM algorithms for solving convex optimization problems involving compositions with linear operators. Besides this, we formulate a variant of the ADMM algorithm that is able to handle convex optimization problems involving an additional smooth function in its objective, and which is evaluated through its gradient. Moreover, in each iteration, we allow the use of variable metrics, while the investigations are carried out in the setting of infinite-dimensional Hilbert spaces. This algorithmic scheme is investigated from the point of view of its convergence properties.

We derive and analyse a new variant of the iteratively regularized Landweber iteration, for solving linear and nonlinear ill-posed inverse problems. The method takes into account training data, which are used to estimate the interior of a black box, which is used to define the iteration process. We prove convergence and stability for the scheme in infinite dimensional Hilbert spaces. These theoretical results are complemented by several numerical experiments for solving linear inverse problems for the Radon transform and a nonlinear inverse problem for Schlieren tomography.

The possibilities of exploiting the special structure of d.c. programs, which consist of optimising the difference of convex functions, are currently more or less limited to variants of the DCA proposed by Pham Dinh Tao and Le Thi Hoai An in 1997. These assume that either the convex or the concave part, or both, are evaluated by one of their subgradients. In this paper we propose an algorithm which allows the evaluation of both the concave and the convex part by their proximal points. Additionally, we allow a smooth part, which is evaluated via its gradient. In the spirit of primal-dual splitting algorithms, the concave part might be the composition of a concave function with a linear operator, which are, however, evaluated separately. For this algorithm we show that every cluster point is a solution of the optimisation problem. Furthermore, we show the connection to the Toland dual problem and prove a descent property for the objective function values of a primal-dual formulation of the problem. Convergence of the iterates is shown if this objective function satisfies the Kurdyka–Łojasiewicz property. In the last part, we apply the algorithm to an image processing model.

Hedy Attouch (University of Montpellier) Relaxed inertial proximal algorithms for monotone inclusions In a Hilbert space H , given A a maximally monotone operator acting on H , we study the convergence properties of a general class of relaxed inertial proximal algorithms. This study aims to extend to the case of monotone inclusions the acceleration techniques initially introduced by Nesterov in the case of convex minimization. As a guideline, we use the interpretation of these algorithms as temporal discretized versions of inertial dynamical systems with vanishing damping. In the general monotone case, the relaxed form of the proximal algorithms plays a central role. It comes naturally with the regularization of the operator A by its Yosida approximation with a variable parameter, a technique recently introduced by Attouch-Peypouquet for a particular class of inertial proximal algorithms. Our study provides an algorithmic version of the convergence results obtained by Attouch-Cabot in the case of continuous dynamical systems. Then, we specialize our study on nonsmooth convex optimization, and show fast convergence properties of these algorithms. We also present new accelerated proximal algorithms based on inertial gradient dynamics which have been rescaled in time. In doing so, we improve and obtain a dynamic interpretation of the seminal papers of Güler on the convergence rate of the proximal methods for convex optimization. Sebastian Banert (KTH Royal Institute of Technology) How to accelerate convex optimisation with machine learning In this talk, we are going to present some ideas how to design algorithms for convex optimisation with possibly nonsmooth functions and choose optimal parameters for them with the help of recent deep learning techniques. Erwin Schrödinger Institute Boltzmanngasse 9/2, A-1090 Vienna, Austria http://www.esi.ac.at/ The main point of this talk will be that one still can get convergence guarantees for the neural networks resulting from this procedure. We will demonstate their performance in variational regularisation of inverse problems in imaging. This talk will present joint work with Axel Ringh, Jonas Adler, Jevgenija Rudzusika, Johan Karlsson, and Ozan Öktem. Amir Beck (Tel-Aviv University) On the convergence to stationary points of deterministic and randomized feasible descent directions methods We study the class of nonsmooth nonconvex problems in which the objective is to minimize the difference between a continuously differentiable function (possibly nonconvex) and a convex (possibly nonsmooth) function over a convex polytope. This general class contains many types of problems, including difference of convex functions (DC) problems, and as such, can be used to model a variety of applications. Our goal is to attain a point satisfying the stationarity necessary condition, defined as the lack of feasible descent directions. In this work we prove that stationarity in our model can be characterized by a finite number of directions that positively span the set of feasible directions. We then use the latter to develop two methods, one deterministic and one random, whose accumulation points are proved to be stationary points. Supplementing discussion on positively spanning sets, and corresponding methods to obtain members of such sets, are also presented. Numerical experiments illustrate the appeal of obtaining a stationary point and the advantage of using the random method to do so. Jérôme Bolte (University Toulouse I Capitole) A multi-proximal method for convex composite optimization Composite minimization involves a collection of smooth functions which are aggregated in a nonsmooth manner. We design an algorithm by linearizing each smooth component in accordance with its main curvature. The resulting method, called the multiprox method, consists in solving successively simple problems (e.g. constrained quadratic problems) which can also feature some proximal operators. To study the complexity and the convergence of this method we introduce a new type of convex qualification condition. We also obtain explicit complexity results involving new types of constant terms. A distinctive feature of our approach is to be able to cope with oracles featuring moving constraints. Our method generalizes the proximal Gauss-Newton’s method, the moving balls method or the forward-backward splitting, for which we recover known complexity results or establish new ones. Immanuel Bomze (University of Vienna) Non-convex min-max fractional quadratic problems under quadratic constraints: copositive relaxations We address a min-max problem of fractional quadratic (not necessarily convex) over linear functions on a feasible set described by linear and (not necessarily convex) quadratic functions. We propose a conic reformulation on the cone of completely positive matrices. By relaxation, a doubly non negative conic formulation is used to provide lower bounds with evidence of very small gaps. It is known that in many solvers using Branch and Bound the optimal solution is obtained in early stages and a heavy computational price is paid in the next iterations to obtain Erwin Schrödinger Institute Boltzmanngasse 9/2, A-1090 Vienna, Austria page 2 the optimality certificate. To reduce this effort, tight lower bounds are crucial. We will show empirical evidence that lower bounds provided by the copositive relaxation are able to substantially speed up a well known solver in obtaining the optimality certificate. This is joint work with Paula Amaral (Univ. Nova de Lisboa, Portugal). Coralia Carţiş (University of Oxford) Dimensionality reduction techniques for global optimization We show that the scalability challenges of Global Optimisation (GO) algorithms can be overcome for functions with low effective dimensionality, which are constant along certain linear subspaces. Such functions can often be found in applications, for example, in hyper-parameter optimization for neural networks, heuristic algorithms for combinatorial optimization problems and complex engineering simulations. We propose the use of random subspace embeddings within a(ny) global minimisation algorithm, extending the approach in Wang et al (2013). Using tools from random matrix theory and conic integral geometry, we investigate the success rates of our lowdimensional embeddings of the original problem, in both a static and adaptive formulation, and show their independence on the (large) ambient dimension of the problem. We illustrate our algorithmic proposals and theoretical findings numerically, using state of the art global solvers. This work is joint with Adilet Otemissov (Turing Institute, London and Oxford University). Volkan Cevher (École Polytechnique Fédérale de Lausanne) Storage optimal semidefinite programming Semidefinite convex optimization problems often have low-rank solutions that can be represented with O(p)storage. However, semidefinite programming methods require us to store the matrix decision variable with size O(p2), which prevents the application of virtually all convex methods at large scale. Indeed, storage, not arithmetic computation, is now the obstacle that prevents us from solving large-scale optimization problems. A grand challenge in contemporary optimization is therefore to design storage-optimal algorithms that provably and reliably solve large-scale optimization problems in key scientific and engineering applications. An algorithm is called storage optimal if its working storage is within a constant factor of the memory required to specify a generic problem instance and its solution. So far, convex methods have completely failed to satisfy storage optimality. As a result, the literature has largely focused on storage optimal non-convex methods to obtain numerical solutions. To this end, my talk introduces a new convex optimization algebra to obtain numerical solutions to semidefinite programs with a low-rank matrix streaming model. This streaming model provides us an opportunity to integrate sketching as a new tool for developing storage optimal convex optimization methods that go beyond semidefinite programming to more general convex templates. We then propose a practical inexact augmented Lagrangian method for non-convex problems with nonlinear constraints and contrast this approach to the convex one. We characterize the total computational complexity of the non-convex method subject to a verifiable geometric condition. Erwin Schrödinger Institute Boltzmanngasse 9/2, A-1090 Vienna, Austria page 3 Patrick Combettes (North Carolina State University) Between subdifferentials and monotone operators We discuss various aspects of the gap between subdifferentials and monotone operators as well as the necessity of the general theory of monotone operators in convex optimization. Aris Daniilidis (University of Chile) Self-contracted curves and extensions The class of self-contracted curves encompasses all (smooth) curves that are orbits of the gradient flow of a smooth quasiconvex function, all (absolutely continuous) curves that that orbits of the subgradient flow of a continuous convex function and all polygonal curves that are obtained by the proximal sequence on a convex continuous function. Self-contracted curves enjoy a simple metric definition and have been the object of the study of several mathematicians working on different domains. In this talk we shall present main ideas, extensions and consequences in numerical optimization. Minh Dào (University of Newcastle) Adaptive Douglas-Rachford splitting algorithm and applications The Douglas–Rachford algorithm is a classical and powerful splitting method for minimizing the sum of two convex functions and, more generally, finding a zero of the sum of two maximally monotone operators. Although this algorithm has been well understood when the involved operators are monotone or strongly monotone, the convergence theory for weakly monotone settings is far from being compl

In this paper, we approach the problem of finding the zeros of the sum of a maximally monotone operator and a monotone and Lipschitz continuous one in a real Hilbert space via an implicit forward-backward-forward dynamical system with nonconstant relaxation parameters and stepsizes of the resolvents. Besides proving existence and uniqueness of strong global solutions for the differential equation under consideration, we show weak convergence of the generated trajectories and, under strong monotonicity assumptions, strong convergence with exponential rate. In the particular setting of minimizing the sum of a proper, convex and lower semicontinuous function with a smooth convex one, we provide a rate for the convergence of the objective function along the ergodic trajectory to its minimum value.

In this paper, we are concerned with solving monotone inclusion problems expressed by the sum of a set-valued maximally monotone operator with a single-valued maximally monotone one and the normal cone to the nonempty set of zeros of another set-valued maximally monotone operator. Depending on the nature of the single-valued operator, we propose two iterative penalty schemes, both addressing the set-valued operators via backward steps. The single-valued operator is evaluated via a single forward step if it is cocoercive, and via two forward steps if it is monotone and Lipschitz continuous. The latter situation represents the starting point for dealing with complexly structured monotone inclusion problems from algorithmic point of view.

The backward-backward algorithm is a tool for finding minima of a regularization of the sum of two convex functions in Hilbert spaces. We generalize this setting to Hadamard spaces and prove the convergence of an error-tolerant version of the backward-backward method.