The non-convex nature of trained neural networks has created significant obstacles in their incorporation into optimization models. In this context, Anderson et al. (2020) provided a framework to obtain the convex hull of the graph of a piecewise linear convex activation function composed with an affine function; this effectively convexifies activations such as the ReLU together with the affine transformation that precedes it. In this article, we contribute to this line of work by developing a recursive formula that yields a tight convexification for the composition of an activation with an affine function for a wide scope of activation functions, namely, convex or “S-shaped”. Our approach can be used to efficiently compute separating hyperplanes or determine that none exists in various settings, including non-polyhedral cases.
We study a class of two-stage stochastic programs, namely, those with fixed recourse matrix and fixed costs, and linear second stage. We show that, under mild assumptions, the problem can be solved with just one scenario, which we call an “optimal scenario.” Such a scenario does not have to be unique and may fall outside the support of the underlying distribution. Although finding an optimal scenario in general might be hard, we show that the result can be particularly useful in the case of stochastic optimization problems with contextual information, where the goal is to optimize the expected value of a certain function given some contextual information (e.g., previous demand, customer type, etc.) that accompany the main data of interest. The contextual information allows for a better estimation of the quantity of interest via machine learning methods. We focus on a class of learning methods—sometimes called in the literature decision-focused learning—that integrate the learning and optimization procedures by means of a bilevel optimization formulation, which determines the parameters for pointwise forecasts. By using the optimal scenario result, we prove that when such models are applied to the class of contextual two-stage problems considered in this paper, the pointwise forecasts computed from the bilevel optimization formulation actually yield asymptotically the best approximation of an optimal scenario within the modeler’s pre-specified set of parameterized forecast functions. Numerical results conducted with inventory problems from the literature (with synthetic data) as well as a bike-sharing problem with real data demonstrate that the proposed approach performs well when compared to benchmark methods from the literature.
We consider minimizing finite-sum objective functions via Hessian-averaging based subsampled Newton methods. These methods allow for gradient inexactness and have fixed per-iteration Hessian approximation costs. The recent work (Na et al. 2023) demonstrated that Hessian averaging can be utilized to achieve fast 𝒪( √(log kk)) local superlinear convergence for strongly convex functions in high probability, while maintaining fixed per-iteration Hessian costs. These methods, however, require gradient exactness and strong convexity, which poses challenges for their practical implementation. To address this concern we consider Hessian-averaged methods that allow gradient inexactness via norm condition based adaptive-sampling strategies. Furthermore, to better control the error in the subsampled Hessian approximations, we utilize Hessian averaging with deterministic cyclic sampling techniques instead of random sampling, which leads to fast local superlinear convergence. We develop a comprehensive convergence theory, including global linear and sublinear convergence rates for strongly convex and nonconvex functions, respectively. Additionally, we establish an improved local superlinear convergence rate of 𝒪( 1k) . Our analysis introduces novel techniques that differ from previous probabilistic approaches. We investigate the performance of these methods on logistic regression problems, demonstrating significant improvements in convergence over similar Hessian-averaging methods that utilize stochastic sampling.
We develop a preference elicitation method for a Von Neumann-Morgenstern (VNM)-type decision-maker from pairwise comparison data in the presence of response errors. We apply the maximum likelihood estimation (MLE) method to jointly elicit the non-parametric systematic VNM utility function and the scale parameter of the response error, assuming a Gumbel distribution. We incorporate structural preference information known in advance about the decision-maker's risk attitude through linear constraints on the utility function, including monotonicity, concavity, and Lipschitz continuity. Under discretely distributed lotteries, the resulting MLE problem can be reformulated as a convex program. We derive finite-sample error bounds between the MLE and the true parameters, and establish quantitative convergence of the MLE-based VNM utility function to the true utility function in the sense of the Kolmogorov distance under some conditions on the lotteries. These conditions may have potential applications in the design of efficient lotteries for preference elicitation. We further show that the optimization problem maximizing the expected MLE-based VNM utility is robust against the response error and estimation error in a probabilistic sense. Numerical experiments in a portfolio optimization application illustrate and support the theoretical results.
Bilevel optimization has garnered growing interest over the past decade. However, little attention has been paid to detecting and dealing with unboundedness in these problems, with most research assuming a bounded high-point relaxation. In this paper, we address unboundedness in bilevel and multilevel optimization by studying its computational complexity. We show that deciding whether an optimistic linear bilevel problem is unbounded is strongly NP-complete, even without coupling constraints. Furthermore, we extend the hardness result to the linear multilevel case, by showing that for each extra level added, the decision problem of checking unboundedness moves up a level in the polynomial hierarchy. Deciding unboundedness of a mixed-integer multilevel problem is shown to be one level higher in the polynomial complexity hierarchy than the decision problem for linear multilevel problem with the same number of levels. Finally, we introduce two algorithmic approaches to determine whether a linear bilevel problem is unbounded and, if so, return a certificate of unboundedness. This certificate consists of a direction of unboundedness and corresponding bilevel feasible point. We present a proof of concept of these algorithmic approaches on some relevant examples, and provide a brief computational comparison.
This article probes the origins of the Column Generation technique. It begins with Kantorovich’s classic 1939 work, correcting widespread misconceptions about his contributions to the Cutting Stock Problem. It then brings to light Kantorovich and Zalgaller’s lesser-known 1951 book, which is revealed to contain a complete Column Generation algorithm. The article also places these contributions in the context of the turbulent USSR’s political and ideological environment, essential for a deeper understanding of their significance.
As a foundation for optimization, convexity is useful beyond the classical settings of Euclidean and Hilbert space. The broader arena of nonpositively curved metric spaces, which includes manifolds like hyperbolic space, as well as metric trees and more general CAT(0) cubical complexes, supports primal tools like proximal operations for geodesically convex functions. However, the lack of linear structure in such spaces complicates dual constructions like subgradients. To address this hurdle, we introduce a new type of subgradient for functions on Hadamard spaces, based on Busemann functions. Our notion supports generalizations of classical stochastic and incremental subgradient methods, with guaranteed complexity bounds. We illustrate with subgradient algorithms for p-mean problems in general Hadamard spaces, in particular computing medians in BHV tree space.
We apply a recently developed framework for analyzing the convergence of stochastic algorithms to the general problem of large-scale nonconvex composite optimization more generally, and nonconvex likelihood maximization in particular. Our theory is demonstrated on a stochastic gradient descent algorithm for determining the electron density of a molecule from random samples of its scattering amplitude. Numerical results on an idealized synthetic example provide a proof of concept. The algorithm we use is just one of a wide range of possibilities, all of which can be formulated abstractly as random function iterations. Our framework provides a basis for evaluating and comparing different numerical strategies. While this case study is very specific, it shares a structure that transfers easily to many problems of current interest, particularly in machine learning.
An interior-point algorithm framework is proposed, analyzed, and tested for solving nonlinearly constrained continuous optimization problems. The main setting of interest is when the objective and inequality constraint functions may be nonlinear and/or nonconvex, and when constraint values and derivatives are tractable to compute, but objective function values and derivatives can only be estimated. The algorithm is intended primarily for a setting that is similar for stochastic-gradient methods for unconstrained optimization, i.e., the setting when stochastic-gradient estimates are available and employed in place of gradients, and when no objective function values (nor estimates of them) are employed. This is achieved by the interior-point framework having a single-loop structure rather than the nested-loop structure that is typical of contemporary interior-point methods. Convergence guarantees for the framework are provided both for deterministic and stochastic settings. Numerical experiments show that the algorithm yields good performance on a large set of test problems.
Polynomial optimization problems (POPs) can be reformulated as geometric convex conic programs, as shown by Kim, Kojima, and Toh (SIOPT 30:1251-1273, 2020), though such formulations remain NP-hard. In this work, we prove that several well-known relaxations can be unified under a common polyhedral-SDP framework, which arises by approximating the intractable cone by tractable intersections of polyhedral cones with the positive semidefinite matrix cone. Although effective in providing tight lower bounds, these relaxations become computationally expensive as the number of variables and constraints grows at the rate of Ω (n^2τ) with the relaxation order τ . To address this challenge, we propose RiNNAL-POP, a low-rank augmented Lagrangian method (ALM) tailored to solve large-scale polyhedral-SDP relaxations of POPs. To efficiently handle the Ω (n^2τ) nonnegativity and consistency constraints, we design a tailored projection scheme whose computational cost scales linearly with the number of variables. In addition, we identify a hidden facial structure in the polyhedral–SDP relaxation, which enables us to eliminate a large number of linear constraints by restricting the matrix variable to affine subspaces corresponding to exposed faces of the semidefinite cone. The latter enables us to efficiently solve the factorized ALM subproblems over the affine subspaces. At each ALM iteration, we additionally carry out a single projected gradient step with respect to the original matrix variable to automatically adjust the rank and escape from spurious local minima when necessary. We also extend our RiNNAL-POP algorithmic framework to solve moment-SOS relaxations of POPs. Extensive numerical experiments on various benchmark problems demonstrate the robustness and efficiency of RiNNAL-POP in solving large-scale polyhedral-SDP relaxations.
The notion of regular pair (A, B) for two nonempty closed convex subsets A and B of a Hilbert space ℋ was introduced by Borwein and Bauschke in 1993 to ensure convergence (in norm) of the alternating projection method to some point of the best approximation set. In 2022, De Bernardi and Miglierina showed that regularity of the pair (A, B) guarantees, additionally, the convergence for any variational perturbation of the alternating projection method, provided the corresponding best approximation sets are bounded. In this work, we show that the converse assertion is also true. Moreover, this converse assertion holds without requiring the best approximation sets to be bounded.
We study a class of multi-stage stochastic programs, which incorporate modeling features from Markov decision processes (MDPs). This class includes structured MDPs with continuous action and state spaces. We extend policy graphs to include decision-dependent uncertainty for one-step transition probabilities as well as a limited form of statistical learning. We focus on the expressiveness of our modeling approach, illustrating ideas with a series of examples of increasing complexity. As a solution method, we develop new variants of stochastic dual dynamic programming, including approximations to handle non-convexities.
We consider the problem of minimizing a sparse nonconvex quadratic function over the unit hypercube. By developing an extension of the Reformulation-Linearization Technique (RLT) to continuous quadratic sets, we propose a novel second-order cone (SOC) representable relaxation for this problem. By exploiting the sparsity of the quadratic function, we establish a sufficient condition under which the convex hull of the feasible region of the lifted quadratic program is SOC-representable. While the proposed formulation may be of exponential size in general, we identify additional structural conditions that guarantee the existence of a polynomial-size SOC-representable formulation, which can be constructed in polynomial time. Under these conditions, the optimal value of the nonconvex quadratic program coincides with that of a polynomial-size second-order cone program. Our results serve as a starting point for bridging the gap between the Boolean quadric polytope of sparse problems and its continuous counterpart.
We consider the sensor network localization problem, which is closely related to multidimensional scaling and Euclidean distance matrix completion. Given a ground truth configuration of n points in ℝ^ℓ , we observe a subset of the pairwise distances and aim to recover the underlying configuration (up to rigid transformations). We show with a simple counterexample that the associated optimization problem is nonconvex and may admit spurious local minimizers, even when all distances are known. Yet, inspired by numerical experiments, we argue that all second-order critical points become global minimizers when the problem is relaxed by optimizing over configurations in dimension k> ℓ . Specifically, we show this for two settings, both when all pairwise distances are known: (1) for arbitrary ground truth points, and k= O(√(ℓ n)) , and: (2) for isotropic random ground truth points, and k= O(ℓ + log n) . To prove these results, we identify and exploit key properties of the linear map which sends inner products to squared distances.
Recent advances in convex optimization have leveraged computer-assisted proofs to develop optimized first-order methods that improve over classical algorithms. However, each optimized method is specially tailored for a particular problem setting, and it is a well-documented challenge to extend optimized methods to other settings due to their highly bespoke design and analysis. We provide a framework that derives optimized methods for composite optimization directly from those for unconstrained smooth optimization. The derived methods naturally extend the original methods, analogous to how proximal gradient descent extends gradient descent. The key to our result is certain algebraic identities that—by leveraging a common structure of optimized methods— provide a unified way of extending convergence analyses from unconstrained to composite settings. As concrete examples, we apply our framework to establish (1) the phenomenon of stepsize acceleration for proximal gradients descent; (2) a convergence rate for the proximal optimized gradient method [2] which is faster than FISTA [3]; (3) a new method that improves the state-of-the-art rate for minimizing gradient norm in the composite setting.
We study the general integer programming (IP) problem of optimizing a separable convex function over the integer points of a polytope: min{f(x) | Ax= b, l≤x≤u, x∈ℤ^n} . The number of variables n is a variable part of the input, and we consider the regime where the constraint matrix A has small coefficients ‖ A‖ _∞ and small primal or dual treedepth td _P(A) or td _D(A) , respectively. Equivalently, we consider block-structured matrices, in particular n-fold, tree-fold, 2-stage and multi-stage matrices. We ask about the possibility of near-linear algorithms in the general case of (non-linear) separable convex functions. The techniques of previous works for the linear case are inherently limited to it; in fact, no strongly-polynomial algorithm may exist due to a simple unconditional information-theoretic lower bound of n log‖u-l‖ _∞ , where l, u are the vectors of lower and upper bounds. Our first result is that with parameters td _P(A) and ‖ A‖ _∞ , this lower bound can be matched (up to dependency on the parameters). Second, with parameters td _D(A) and ‖ A‖ _∞ , the situation is more involved, and we design an algorithm with complexity g( td _D(A), ‖ A‖ _∞ ) n log n log‖u-l‖ _∞ where g is some computable function. We conjecture that a stronger lower bound is possible in this regime, and our algorithm is in fact optimal. Our algorithms combine ideas from scaling, proximity, and sensitivity of integer programs, together with a new dynamic data structure allowing fast sparse updates.
We analyze fast diagonal methods for simple bilevel programs. Guided by the analysis of the corresponding continuous-time dynamics, under a mild general assumption we provide convergence rates for the inner residual and upper bounds for the outer residual along an ergodic sequence. As the key point of our work, under geometric assumptions for the inner function—namely, a weaker condition attributed to Attouch and Czarnecki and a stronger Hölderian error bound—we provide a unified convergence analysis, which yields explicit last-iterate convergence rates and guarantees weak convergence to a solution of the bilevel problem. In particular, we improve and extend recent results on accelerated schemes, offering novel insights into the trade-offs between geometry, regularization decay, and algorithmic design. Numerical experiments illustrate the advantages of more flexible methods and support our theoretical findings.
In this paper, we consider the nonlinear constrained optimization problem (NCP) with a constraint set 𝒦:={x ∈𝒳: c(x) = 0} , where 𝒳 is a closed convex subset of ℝ^n . We propose an exact penalty approach, named constraint dissolving approach, that transforms (NCP) into its corresponding constraint dissolving problem (CDP). The transformed problem (CDP) admits 𝒳 as its feasible region with a locally Lipschitz smooth objective function. We prove that (NCP) and (CDP) share the same first-order stationary points, second-order stationary points, second-order sufficient condition (SOSC) points, and strong SOSC points, in a neighborhood of the feasible region 𝒦 . Moreover, we prove that these equivalences extend globally under a particular error bound condition. Therefore, our proposed constraint dissolving approach enables direct implementations of optimization approaches over 𝒳 and inherits their convergence properties for solving problems of the form (NCP). Preliminary numerical experiments illustrate the high efficiency of directly applying existing solvers for optimization over 𝒳 to solve (NCP) through (CDP). These numerical results further demonstrate the practical potential of our proposed constraint dissolving approach.
Totally equimodular matrices generalize totally unimodular matrices and arise in the context of box-totally dual integral polyhedra. This work further explores the parallels between these two classes and introduces foundational building blocks for constructing totally equimodular matrices. Consequently, we present a decomposition theorem for totally equimodular matrices of full row rank. Building on this decomposition theorem, we prove that simplicial cones whose generators form the rows of a totally equimodular matrix satisfy strong integrality decomposition properties. More precisely, we provide the Hilbert basis for these cones and construct regular unimodular Hilbert triangulations in most cases. We conjecture that cases not covered here do not exist.
