This paper explores Augmented Lagrangian methods for derivative-free continuous optimization under constraints. The theoretical analysis proceeds without assuming the existence of derivatives, and the proofs of these results bypass traditional KKT conditions. Numerical experiments demonstrate the fitting of parameters within river flow models. These models address the estimation of hydraulic parameters by minimizing their dispersion, subject to both differential equations and empirical observations. The experiments are situated within the framework of the CRIAB project, which focuses on the study of dam failure.
This paper introduces novel alternate training procedures for hard-parameter sharing Multi-Task Neural Networks (MTNNs). Traditional MTNN training faces challenges in managing conflicting loss gradients, often yielding sub-optimal performance. The proposed alternate training method updates shared and task-specific weights alternately through the epochs, exploiting the multi-head architecture of the model. This approach reduces computational costs per epoch and memory requirements. Convergence properties similar to those of the classical stochastic gradient method are established. Empirical experiments demonstrate enhanced training regularization and reduced computational demands. In summary, our alternate training procedures offer a promising advancement for the training of hard-parameter sharing MTNNs.
We introduce a notion of self-concordant smoothing for minimizing the sum of two convex functions, one of which is smooth and the other nonsmooth. The key highlight is a natural property of the resulting problem's structure that yields a variable metric selection method and a step length rule especially suited to proximal quasi-Newton algorithms. Also, we efficiently handle specific structures promoted by the nonsmooth term, such as $ \ell _{1} $ & ell;1-regularization and group-lasso penalties. A convergence analysis for the class of proximal quasi-Newton methods covered by our framework is presented. In particular, we obtain guarantees, under standard assumptions, for two algorithms: Prox-N-SCORE (a proximal Newton method) and Prox-GGN-SCORE (a proximal generalized Gauss-Newton method). The latter uses a low-rank approximation of the Hessian inverse, reducing most of the cost of matrix inversion and making it effective for overparameterized machine learning models. Numerical experiments on synthetic and real data demonstrate the efficiency of both algorithms against state-of-the-art approaches. A Julia implementation is publicly available at https://github.com/adeyemiadeoye/SelfConcordantSmoothOptimization.jl.
A block coordinate descent method, is an iterative procedure that solves block-structured optimization problems, where each block of coordinates is subject to its own constraints, by sequentially solving the same problem with respect to each block, while the other coordinates are held fixed. For such problems, we develop new methods that act by solving less constrained and of smaller size subproblems. We propose two methods: The block coordinate descent method of centres and the block coordinate descent proximal method of centres. They have two advantages: From one side, the intermediate problems to solve are unconstrained or at least less constrained, and from an other side, are of smaller size than the original problem. Under regularity properties involving the objective function, we establish optimality conditions on the entire decision variable, based on those established for its block coordinates. These methods generate sequences whose every cluster point satisfies these optimality conditions. Finally, we conclude by numerical examples for comparing our algorithms with existing methods and testing their efficiency.
In this article, with a convex combination of two conjugate gradient (CG) parameters, a novel improved CG parameter is introduced that leads to a new CG method with more appropriate theoretical and computational properties. The convex combination parameter in our proposed CG parameter is obtained such that the search direction satisfies the Dai-Liao conjugacy condition and aligns with the quasi-Newton direction. Then, the corresponding spectral CG method is formulated, having the property that its generated search direction satisfies the sufficient descent property independent of the line search technique. Under some appropriate assumptions, we establish the global convergence of our method for general non-convex functions. Numerical results on solving unconstrained optimization test problems from the CUTEst collection demonstrate the efficiency of our proposed approach in application. Moreover, we also demonstrate the efficiency of our proposed method in solving image restoration problems.
We study multiobjective composite optimization problems (MOCO), where each objective function is the sum of a continuously differentiable term and a possibly non-differentiable convex function. To generalize a technique traditionally applied in scalar optimization to the multiobjective setting, we propose a generalized conditional-gradient method (ConG) also known as the Frank-Wolfe method augmented with a non-monotone line search strategy. Unlike existing approaches, which rely on monotone or Armijo rules, our method incorporates more flexible stepsize procedures, including max-type, moving average, summable, diminishing, and gap-dependent strategies. The algorithm is designed around two tunable components: a classical backtracking rule and a novel adaptive strategy that reuses previous stepsizes, reducing computational cost. We establish asymptotic convergence to Pareto-critical points under mild boundedness assumptions and derive worst-case iteration-complexity bounds of order $ \mathcal {O}(1/\varepsilon <^>2) $ O(1/epsilon 2) when the differentiable components have Lipschitz gradients matching the best known rates for scalar conditional-gradient methods. We also provide explicit bounds on the number of function and gradient evaluations required for each stepsize rule. A key technical contribution is the use of a gap function that measures stationarity without scalarization, serving both as a stopping criterion and the basis of the complexity analysis. Numerical experiments confirm the effectiveness and efficiency of the proposed method on a variety of test problems.
Recently gradient-free optimization methods have become a major tool in reinforcement learning and memory-efficient LLM fine-tuning. Under the standard setting of uniformly bounded noise variance an optimal accelerated algorithm has been derived. However, the assumption of bounded variance is strict and usually is not fulfilled in practice. Therefore, we will relax it, allowing the noise distribution to be heavy-tailed and, thus, broadening the class of problems to be solved. We propose gradient-free algorithms with zeroth-order oracle under adversarial noise with unbounded variance, for non-smooth convex and convex-concave optimization problems. We apply clipping operator to deal with heavy-tailedness and batching to allow efficient computation via parallelization. Our analysis provides asymptotic bounds for such key parameters as iteration complexity, oracle complexity and maximal adversarial noise level.
In this paper, we propose a gradient-type method for solving a quadratic separable knapsack problem, which is simpler than the existing methods and competitive in practice. The algorithm uses a smooth approximation of the dual function of the knapsack problem. Partial convergence and complexity results of the algorithm are presented. Moreover, to demonstrate its efficiency, illustrative computational results are reported.
In this paper, we investigate a warped iteration for solving the nonmonotone inclusions constituted by the sum of two operators, where one is cocoercive and the other is possibly nonmonotone. Under appropriate conditions on the parameters, the weak convergence of the sequence of iteration is derived, with worst-case rates of $ o(k<^>{-1}) $ o(k-1) in terms of both the discrete velocity and the residual norm. We also give some simple numerical experiments to demonstrate the efficiency of the proposed algorithm.
In this paper we discuss the extension of an RLT-based algorithm for continuous polynomial optimization problems to handle mixed-integer variables. The chosen approach is a direct one, in which the LP relaxations to be solved at the nodes of the branch-and-bound tree are replaced with MILP relaxations and, therefore, the additional burden caused by the discrete variables is taken care of by the auxiliary MILP solver. One of the main advantages of this approach is that the resulting algorithm inherits all the strengths of the auxiliary MILP solver. We conduct a computational analysis in which we focus on the impact of a number of choices that must be made for the resulting algorithm to be effective at providing good lower and upper bounds upon termination. All the analyses are carried out within RAPOSa, a state-of-the-art global solver for polynomial optimization.
NashOpt is an open-source Python library for computing and designing generalized Nash equilibria (GNEs) in noncooperative games with shared constraints and real-valued decision variables. The library exploits the joint Karush-Kuhn-Tucker (KKT) conditions of all players to handle both general nonlinear GNEs and linear-quadratic games, including their variational versions. Nonlinear games are solved via nonlinear least-squares formulations, relying on JAX for automatic differentiation. Linear-quadratic GNEs are reformulated as mixed-integer linear programs, enabling efficient computation of multiple equilibria. The framework also supports inverse-game and Stackelberg game-design problems. The capabilities of NashOpt are demonstrated through several examples, including noncooperative game-theoretic control problems of linear quadratic regulation and model predictive control. The library is available at https://github.com/bemporad/nashopt
In this paper we study a nonconvex-strongly-concave constrained minimax problem. Specifically, we propose a first-order augmented Lagrangian method for solving it, whose subproblems are nonconvex-strongly-concave unconstrained minimax problems and suitably solved by a first-order method developed in this paper that leverages the strong concavity structure. Under suitable assumptions, the proposed method achieves an operation complexity of $ \mathcal {O}(\varepsilon {-3.5}\log \varepsilon {-1}) $ O(epsilon-3.5log epsilon-1), measured in terms of its fundamental operations, for finding an epsilon-KKT solution of the constrained minimax problem, which improves the previous best-known operation complexity by a factor of $ \varepsilon {-0.5} $ epsilon-0.5.
The input parameters of an optimization problem are often affected by uncertainties. Chance constraints are one way to model stochastic uncertainties in the constraints. Typically, solution algorithms for chance-constrained problems require convex functions or discrete distributions. Here, we go one step further and allow non-convexities and continuous distributions. We propose a gradient-based approach to approximately solve joint chance-constrained models. We approximate the original problem by smoothing indicator functions. The smoothed chance constraints are relaxed by penalizing their violation in the objective function. The approximation problem is solved with the Continuous Stochastic Gradient method that is an enhanced version of the stochastic gradient descent and has recently been introduced in the literature. We present a convergence theory for the smoothing and penalty approximations. Under very mild assumptions, our approach is applicable to a wide range of problems. We illustrate its computational efficiency on difficult practical problems in gas networks. The numerical experiments demonstrate that the approach quickly finds nearly feasible solutions for joint chance-constrained problems with non-convex constraint functions and continuous distributions, even for realistically-sized instances.
We introduce the Method of Ellipcenters (ME) for unconstrained minimization. At the cost of two gradients per iteration and a line search, we compute the next iterate by setting it as the centre of an elliptical interpolation. The idea behind the ellipse built in each step is to emulate the original level curve of the objective function constrained to a suitable two-dimensional affine space, which is determined by the current iterate and two appropriate gradient vectors. We present the method for general unconstrained minimization and carry out a convergence analysis for the case where the objective function is quadratic. In this context, ME enjoys linear convergence with the rate being at least as good as the linear rate of the steepest descent (gradient) method with optimal step. In our experiments, however, ME was much faster than the gradient method with optimal step size. Moreover, ME seems highly competitive in comparison to several well-established algorithms. The efficiency in terms of both time and number of iterations is stressed even more for ill-conditioned problems.
The Burer-Monteiro factorization has become a powerful tool for solving large-scale semidefinite programs (SDPs), enabling recently developed low-rank solvers to tackle problems previously beyond reach. However, existing methods are typically designed to prioritize scalability over solution accuracy. We introduce the Augmented Mixing Method, a new algorithm that combines the Burer-Monteiro factorization with an inexact augmented Lagrangian framework and a block coordinate descent scheme. Our method emphasizes solving low-dimensional subproblems efficiently and to high precision. Inequality constraints are handled directly, without explicitly maintaining slack variables in the algorithm. A novel dynamic update strategy for the penalty parameter ensures that primal and dual feasibility progress remain balanced. This approach enables our method to compute highly accurate primal-dual solutions, even for large-scale SDPs with over ten million inequality constraints. Despite lacking theoretical convergence guarantees, the Augmented Mixing Method shows strong practical performance with default parameters across a wide range of SDP instances. It often produces more accurate primal-dual solutions than state-of-the-art interior-point methods and scales significantly better. Our open-source Julia implementation is memory-efficient, customizable, and supports arbitrary-precision arithmetic.
Derivative-free optimization (DFO) problems are optimization problems where derivative information is unavailable or extremely difficult to obtain. Model-based DFO solvers have been applied extensively in scientific computing. Powell's NEWUOA (2004) and Wild's POUNDerS (2014) explore the numerical power of the minimal norm Hessian (MNH) model for DFO and contributed to the open discussion on building better models with fewer data to achieve faster numerical convergence. Another decade later, we propose the regional minimal updating (ReMU) models, and extend the previous models into a broader class. This paper shows motivation behind ReMU models, computational details, theoretical and numerical results on particular extreme points and the barycenter of ReMU's weight coefficient region, and the associated KKT matrix error and distance. Novel metrics, such as the truncated Newton step error, are proposed to numerically understand the new models' properties. A new algorithmic strategy, based on iteratively adjusting the ReMU model type, is also proposed, and shows numerical advantages by combining and switching between the barycentric model and the classic least Frobenius norm model in an online fashion.
