In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most 𝒪(d) iterations, where d is the problem’s dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of 𝒪(min{1/k,√(d)/k^1.25}) in terms of the duality gap. This matches the rate of the extragradient method when k = 𝒪(d^2) and is faster when k = (d^2) . These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.
Inspired by Shapiro et al. [74], we consider a stochastic optimal control (SOC) and Markov decision process (MDP) under simultaneous epistemic and aleatoric uncertainties using Bayesian composite risk (BCR) measures. The proposed BCR-SOC/MDP model evaluates the risk of stagewise cost via a two-layer framework: the inner risk measure tackles aleatoric uncertainty conditional on a latent environment parameter, while the outer risk measure deals with the epistemic uncertainty of the inner risk under the Bayesian posterior. The resulting time-varying risk evaluation induced by Bayesian updating enables an information-adaptive risk-sensitive decision framework. Unlike [74], our policies are allowed to depend explicitly on the posterior belief, reflecting that accumulated information about epistemic uncertainty can influence the assessment of future aleatoric uncertainty and, consequently, the decision maker’s actions [79]. The new modeling paradigm subsumes several classical SOC/MDP formulations, including risk-averse and distributionally robust SOC/MDPs as well as partially observed and Bayes-adaptive MDPs, and generates so-called preference robust SOC/MDP models. Moreover, we derive conditions under which the BCR-SOC/MDP model is well-defined, show that finite-horizon BCR-SOC/MDP models can be solved via dynamic programming, and extend the analysis to the infinite-horizon case. Under standard conditions, we establish asymptotic convergence of the optimal values and optimal policies as data accumulate, and provide quantitative error bounds for several representative classes of risk measures. To enhance computational tractability, we develop a hyper-parameter discretization approach for the posterior belief space. Finally, we carry out numerical tests on a spread betting problem and an inventory control problem, demonstrating the effectiveness of the proposed model and numerical schemes.
The single-source unsplittable flow (SSUF) problem asks to send flow from a common source to terminals with unrelated demands, each terminal being served through a single path. The classical SSUF objective is to minimize the violation of some given arc capacities. A seminal result of Dinitz, Garg, and Goemans showed that, whenever a fractional flow exists respecting the capacities, then there is an unsplittable one violating the capacities by at most the maximum demand. Goemans conjectured a natural cost version of the same result, where the unsplittable flow is required to be no more expensive than the fractional one. Intriguingly, there are arguably no non-trivial graph classes for which it is known to hold. We show that a slight weakening of it holds for planar graphs, by exploiting a connection to a highly structured discrepancy problem. Moreover, our techniques extend to simultaneous upper and lower bounds on the flow values. This affirmatively answers a conjecture of Morell and Skutella for planar SSUF. Finally, we show that our approach can be extended to general (non-planar) graphs with a capacity violation that depends on the genus.
In the Flexible Graph Connectivity (FGC) problem, we are given an undirected multigraph on n vertices with nonnegative edge costs, where each edge is classified as either safe or unsafe. Given integer parameters p and q, the goal in (p, q)-FGC is to purchase a minimum-cost set of edges such that the resulting spanning subgraph remains p-edge-connected after the removal of any set of up to q unsafe edges. Our main contribution is an O(log n) -approximation algorithm based on independent rounding, improving the previous best approximation ratio of O(q log n) . Central to our approach is a new linear programming formulation of feasible solutions that encodes knapsack cover inequalities as cut-capacity constraints. Unlike prior work, the capacity of an edge in a cut may depend on the partially purchased solution for this cut. We show that the resulting linear program admits a polynomial-time separation oracle. Scaling the fractional solution by (log n) and applying independent rounding yields a feasible integral solution with constant probability; here, we leverage the knapsack cover inequalities to obtain strong concentration bounds for the rounded solution relative to any given partial solution. A key ingredient in both separation and rounding is the use of Karger’s bound on the number of near-minimum cuts. We also extend the (p, q)-FGC problem to model more than two safety tiers and show that our results and techniques extend naturally to this setting, albeit with increased approximation ratios and running times that scale with the number of tiers.
We consider the multilinear polytope, defined as the convex hull of the feasible region of a lifted binary polynomial optimization problem. We define a relaxation in an extended space for this polytope, which we call the complete edge relaxation. The complete edge relaxation is stronger than several well-known relaxations of the multilinear polytope, including the standard linearization, the flower relaxation, and the intersection of all possible recursive McCormick relaxations. In addition, for fixed-degree binary polynomial optimization problems, the case of primary practical interest, the complete edge relaxation is of polynomial size and is computationally efficient in practice. We prove that the complete edge relaxation is an extension of the multilinear polytope if and only if the corresponding hypergraph is α -acyclic, the most general type of hypergraph acyclicity. This is in stark contrast with the widely-used standard linearization, which describes the multilinear polytope if and only if the hypergraph is Berge-acyclic, the most restrictive type of hypergraph acyclicity. Finally, we introduce a new class of facet-defining inequalities for the multilinear polytope of α -cycles of length three, which serve as the generalization of the well-known triangle inequalities for the Boolean quadric polytope.
The non-convex nature of trained neural networks has created significant obstacles in their incorporation into optimization models. In this context, Anderson et al. (2020) provided a framework to obtain the convex hull of the graph of a piecewise linear convex activation function composed with an affine function; this effectively convexifies activations such as the ReLU together with the affine transformation that precedes it. In this article, we contribute to this line of work by developing a recursive formula that yields a tight convexification for the composition of an activation with an affine function for a wide scope of activation functions, namely, convex or “S-shaped”. Our approach can be used to efficiently compute separating hyperplanes or determine that none exists in various settings, including non-polyhedral cases.
We study the budgeted versions of the well known matching, matroid independent set, and matroid intersection problems. While all problems admit polynomial-time approximation schemes (PTAS) [Berger et al. (Math. Programming, 2011), Chekuri, Vondrák and Zenklusen (SODA 2011)], it has been an intriguing open question whether these problems admit an efficient PTAS (EPTAS). In this paper, we answer this question affirmatively, by presenting an EPTAS for budgeted matching, budgeted matroid independent set, and budgeted matroid intersection. As we recently showed that budgeted matroid independent set and budgeted matroid intersection do not admit a fully PTAS (FPTAS), this paper resolves the complexity status of the two problems. The running times of previous schemes for these problems are dominated by exhaustive enumeration over the O( 1/ε) most profitable elements in an optimal solution, where ε is the accuracy parameter. We replace this enumeration with a new framework of representative sets, in which we enumerate over subsets of a representative set of cardinality that depends only on ε .
We study a class of two-stage stochastic programs, namely, those with fixed recourse matrix and fixed costs, and linear second stage. We show that, under mild assumptions, the problem can be solved with just one scenario, which we call an “optimal scenario.” Such a scenario does not have to be unique and may fall outside the support of the underlying distribution. Although finding an optimal scenario in general might be hard, we show that the result can be particularly useful in the case of stochastic optimization problems with contextual information, where the goal is to optimize the expected value of a certain function given some contextual information (e.g., previous demand, customer type, etc.) that accompany the main data of interest. The contextual information allows for a better estimation of the quantity of interest via machine learning methods. We focus on a class of learning methods—sometimes called in the literature decision-focused learning—that integrate the learning and optimization procedures by means of a bilevel optimization formulation, which determines the parameters for pointwise forecasts. By using the optimal scenario result, we prove that when such models are applied to the class of contextual two-stage problems considered in this paper, the pointwise forecasts computed from the bilevel optimization formulation actually yield asymptotically the best approximation of an optimal scenario within the modeler’s pre-specified set of parameterized forecast functions. Numerical results conducted with inventory problems from the literature (with synthetic data) as well as a bike-sharing problem with real data demonstrate that the proposed approach performs well when compared to benchmark methods from the literature.
We consider minimizing finite-sum objective functions via Hessian-averaging based subsampled Newton methods. These methods allow for gradient inexactness and have fixed per-iteration Hessian approximation costs. The recent work (Na et al. 2023) demonstrated that Hessian averaging can be utilized to achieve fast 𝒪( √(log kk)) local superlinear convergence for strongly convex functions in high probability, while maintaining fixed per-iteration Hessian costs. These methods, however, require gradient exactness and strong convexity, which poses challenges for their practical implementation. To address this concern we consider Hessian-averaged methods that allow gradient inexactness via norm condition based adaptive-sampling strategies. Furthermore, to better control the error in the subsampled Hessian approximations, we utilize Hessian averaging with deterministic cyclic sampling techniques instead of random sampling, which leads to fast local superlinear convergence. We develop a comprehensive convergence theory, including global linear and sublinear convergence rates for strongly convex and nonconvex functions, respectively. Additionally, we establish an improved local superlinear convergence rate of 𝒪( 1k) . Our analysis introduces novel techniques that differ from previous probabilistic approaches. We investigate the performance of these methods on logistic regression problems, demonstrating significant improvements in convergence over similar Hessian-averaging methods that utilize stochastic sampling.
We develop a preference elicitation method for a Von Neumann-Morgenstern (VNM)-type decision-maker from pairwise comparison data in the presence of response errors. We apply the maximum likelihood estimation (MLE) method to jointly elicit the non-parametric systematic VNM utility function and the scale parameter of the response error, assuming a Gumbel distribution. We incorporate structural preference information known in advance about the decision-maker's risk attitude through linear constraints on the utility function, including monotonicity, concavity, and Lipschitz continuity. Under discretely distributed lotteries, the resulting MLE problem can be reformulated as a convex program. We derive finite-sample error bounds between the MLE and the true parameters, and establish quantitative convergence of the MLE-based VNM utility function to the true utility function in the sense of the Kolmogorov distance under some conditions on the lotteries. These conditions may have potential applications in the design of efficient lotteries for preference elicitation. We further show that the optimization problem maximizing the expected MLE-based VNM utility is robust against the response error and estimation error in a probabilistic sense. Numerical experiments in a portfolio optimization application illustrate and support the theoretical results.
Bilevel optimization has garnered growing interest over the past decade. However, little attention has been paid to detecting and dealing with unboundedness in these problems, with most research assuming a bounded high-point relaxation. In this paper, we address unboundedness in bilevel and multilevel optimization by studying its computational complexity. We show that deciding whether an optimistic linear bilevel problem is unbounded is strongly NP-complete, even without coupling constraints. Furthermore, we extend the hardness result to the linear multilevel case, by showing that for each extra level added, the decision problem of checking unboundedness moves up a level in the polynomial hierarchy. Deciding unboundedness of a mixed-integer multilevel problem is shown to be one level higher in the polynomial complexity hierarchy than the decision problem for linear multilevel problem with the same number of levels. Finally, we introduce two algorithmic approaches to determine whether a linear bilevel problem is unbounded and, if so, return a certificate of unboundedness. This certificate consists of a direction of unboundedness and corresponding bilevel feasible point. We present a proof of concept of these algorithmic approaches on some relevant examples, and provide a brief computational comparison.
This article probes the origins of the Column Generation technique. It begins with Kantorovich’s classic 1939 work, correcting widespread misconceptions about his contributions to the Cutting Stock Problem. It then brings to light Kantorovich and Zalgaller’s lesser-known 1951 book, which is revealed to contain a complete Column Generation algorithm. The article also places these contributions in the context of the turbulent USSR’s political and ideological environment, essential for a deeper understanding of their significance.
As a foundation for optimization, convexity is useful beyond the classical settings of Euclidean and Hilbert space. The broader arena of nonpositively curved metric spaces, which includes manifolds like hyperbolic space, as well as metric trees and more general CAT(0) cubical complexes, supports primal tools like proximal operations for geodesically convex functions. However, the lack of linear structure in such spaces complicates dual constructions like subgradients. To address this hurdle, we introduce a new type of subgradient for functions on Hadamard spaces, based on Busemann functions. Our notion supports generalizations of classical stochastic and incremental subgradient methods, with guaranteed complexity bounds. We illustrate with subgradient algorithms for p-mean problems in general Hadamard spaces, in particular computing medians in BHV tree space.
We apply a recently developed framework for analyzing the convergence of stochastic algorithms to the general problem of large-scale nonconvex composite optimization more generally, and nonconvex likelihood maximization in particular. Our theory is demonstrated on a stochastic gradient descent algorithm for determining the electron density of a molecule from random samples of its scattering amplitude. Numerical results on an idealized synthetic example provide a proof of concept. The algorithm we use is just one of a wide range of possibilities, all of which can be formulated abstractly as random function iterations. Our framework provides a basis for evaluating and comparing different numerical strategies. While this case study is very specific, it shares a structure that transfers easily to many problems of current interest, particularly in machine learning.
