In recent years, there has been growing interest in solving linear optimization problems - or more simply "LP" - using first-order methods. The restarted primal-dual hybrid gradient method (PDHG) - together with some heuristic techniques - has emerged as a powerful tool for solving huge-scale LPs. However, the theoretical understanding of it and the validation of various heuristic implementation techniques are still very limited. Existing complexity analyses have relied on the Hoffman constant of the LP KKT system, which is known to be overly conservative, difficult to compute (and hence difficult to empirically validate), and fails to offer insight into instance-specific characteristics of the LP problems. These limitations have limited the capability to discern which characteristics of LP instances lead to easy versus difficult LP. With the goal of overcoming these limitations, in this paper we introduce and develop two purely geometry-based condition measures for LP instances: the "limiting error ratio" and the LP sharpness. We provide new computational guarantees for the restarted PDHG based on these two condition measures. For the limiting error ratio, we provide a computable upper bound and show its relationship with the data instance's proximity to infeasibility under perturbation. For the LP sharpness, we prove its equivalence to the stability of the LP optimal solution set under perturbation of the objective function. We validate our computational guarantees in terms of these condition measures via specially constructed instances. Conversely, our computational guarantees validate the practical efficacy of certain heuristic techniques (row preconditioners and step-size tuning) that improve computational performance in practice. Finally, we present computational experiments on LP relaxations from the MIPLIB dataset that demonstrate the promise of various implementation strategies.

The Frank-Wolfe method has become increasingly useful in statistical and machine learning applications, due to the structure-inducing properties of the iterates, and especially in settings where linear minimization over the feasible set is more computationally efficient than projection. In the setting of Empirical Risk Minimization -- one of the fundamental optimization problems in statistical and machine learning -- the computational effectiveness of Frank-Wolfe methods typically grows linearly in the number of data observations $n$. This is in stark contrast to the case for typical stochastic projection methods. In order to reduce this dependence on $n$, we look to second-order smoothness of typical smooth loss functions (least squares loss and logistic loss, for example) and we propose amending the Frank-Wolfe method with Taylor series-approximated gradients, including variants for both deterministic and stochastic settings. Compared with current state-of-the-art methods in the regime where the optimality tolerance $\varepsilon$ is sufficiently small, our methods are able to simultaneously reduce the dependence on large $n$ while obtaining optimal convergence rates of Frank-Wolfe methods, in both the convex and non-convex settings. We also propose a novel adaptive step-size approach for which we have computational guarantees. Last of all, we present computational experiments which show that our methods exhibit very significant speed-ups over existing methods on real-world datasets for both convex and non-convex binary classification problems.

There has been a recent surge in development of first-order methods (FOMs) for solving huge-scale linear programming (LP) problems. The attractiveness of FOMs for LP stems in part from the fact that they avoid costly matrix factorization computation. However, the efficiency of FOMs is significantly influenced - both in theory and in practice - by certain instance-specific LP condition measures. Xiong and Freund recently showed that the performance of the restarted primal-dual hybrid gradient method (PDHG) is predominantly determined by two specific condition measures: LP sharpness and Limiting Error Ratio. In this paper we examine the relationship between these two measures, particularly in the case when the optimal solution is unique (which is generic - at least in theory), and we present an upper bound on the Limiting Error Ratio involving the reciprocal of the LP sharpness. This shows that in LP instances where there is a dual nondegenerate optimal solution, the computational complexity of restarted PDHG can be characterized solely in terms of LP sharpness and the distance to optimal solutions, and simplifies the theoretical complexity upper bound of restarted PDHG for these instances.

We present and analyze a new generalized Frank–Wolfe method for the composite optimization problem (P): min_x∈ℝ^n f(𝖠 x) + h(x) , where f is a θ -logarithmically-homogeneous self-concordant barrier, 𝖠 is a linear operator and the function h has a bounded domain but is possibly non-smooth. We show that our generalized Frank–Wolfe method requires O((δ _0 + θ + R_h)ln (δ _0) + (θ + R_h)^2/ε ) iterations to produce an ε -approximate solution, where δ _0 denotes the initial optimality gap and R_h is the variation of h on its domain. This result establishes certain intrinsic connections between θ -logarithmically homogeneous barriers and the Frank–Wolfe method. When specialized to the D-optimal design problem, we essentially recover the complexity obtained by Khachiyan (Math Oper Res 21 (2): 307–320, 1996) using the Frank–Wolfe method with exact line-search. We also study the (Fenchel) dual problem of (P), and we show that our new method is equivalent to an adaptive-step-size mirror descent method applied to the dual problem. This enables us to provide iteration complexity bounds for the mirror descent method despite the fact that the dual objective function is non-Lipschitz and has unbounded domain. In addition, we present computational experiments that point to the potential usefulness of our generalized Frank–Wolfe method on Poisson image de-blurring problems with TV regularization, and on simulated PET problem instances.

The stochastic Frank-Wolfe method has recently attracted much general interest in the context of optimization for statistical and machine learning due to its ability to work with a more general feasible region. However, there has been a complexity gap in the guaranteed convergence rate for stochastic Frank-Wolfe compared to its deterministic counterpart. In this work, we present a new generalized stochastic Frank-Wolfe method which closes this gap for the class of structured optimization problems encountered in statistical and machine learning characterized by empirical loss minimization with a certain type of ``linear prediction'' property (formally defined in the paper), which is typically present loss minimization problems in practice. Our method also introduces the notion of a ``substitute gradient'' that is a not-necessarily-unbiased sample of the gradient. We show that our new method is equivalent to a particular randomized coordinate mirror descent algorithm applied to the dual problem, which in turn provides a new interpretation of randomized dual coordinate descent in the primal space. Also, in the special case of a strongly convex regularizer our generalized stochastic Frank-Wolfe method (as well as the randomized dual coordinate descent method) exhibits linear convergence. Furthermore, we present computational experiments that indicate that our method outperforms other stochastic Frank-Wolfe methods consistent with the theory developed herein.

Motivated by recent work of Renegar (A framework for applying subgradient methods to conic optimization problems, arXiv:1503.02611v2, 2015) we present new computational methods and associated computational guarantees for solving convex optimization problems using first-order methods. Our problem of interest is the general convex optimization problem \(f^* = \min _{x \in Q} f(x)\), where we presume knowledge of a strict lower bound \(f_{\mathrm{slb}}< f^*\). [Indeed, \(f_{\mathrm{slb}}\) is naturally known when optimizing many loss functions in statistics and machine learning (least-squares, logistic loss, exponential loss, total variation loss, etc.) as well as in Renegar’s transformed version of the standard conic optimization problem arXiv:1503.02611v2, 2015; in all these cases one has \(f_{\mathrm{slb}}= 0 < f^*\).] We introduce a new functional measure called the growth constant G for \(f(\cdot )\), that measures how quickly the level sets of \(f(\cdot )\) grow relative to the function value, and that plays a fundamental role in the complexity analysis. When \(f(\cdot )\) is non-smooth, we present new computational guarantees for the Subgradient Descent Method and for smoothing methods, that can improve existing computational guarantees in several ways, most notably when the initial iterate \(x^0\) is far from the optimal solution set. When \(f(\cdot )\) is smooth, we present a scheme for periodically restarting the Accelerated Gradient Method that can also improve existing computational guarantees when \(x^0\) is far from the optimal solution set, and in the presence of added structure we present a scheme using parametrically increased smoothing that further improves the associated computational guarantees.

We present new results for the Frank-Wolfe method (also known as the conditional gradient method). We derive computational guarantees for arbitrary step-size sequences, which are then applied to various step-size rules, including simple averaging and constant step-sizes. We also develop step-size rules and computational guarantees that depend naturally on the warm-start quality of the initial (and subsequent) iterates. Our results include computational guarantees for both duality/bound gaps and the so-called FW gaps. Lastly, we present complexity bounds in the presence of approximate computation of gradients and/or linear optimization subproblem solutions.

Designing phononic crystals by creating frequency bandgaps is of particular interest in the engineering of elastic and acoustic microstructured materials. Mathematically, the problem of optimizing the frequency bandgaps is often nonconvex, as it requires the maximization of the higher indexed eigenfrequency and the minimization of the lower indexed eigenfrequency. A novel algorithm [1] has been previously developed to reformulate the original nonlinear, nonconvex optimization problem to an iteration-specific semidefinite program (SDP). This algorithm separates two consecutive eigenvalues — effectively maximizing bandgap (or bandwidth) — by separating the gap between two orthogonal subspaces, which are comprised columnwise of “important” eigenvectors associated with the eigenvalues being bounded. By doing so, we avoid the need of computation of eigenvalue gradient by computing the gradient of affine matrices with respect to the decision variables. In this work, we propose an even more efficient algorithm based on linear programming (LP). The new formulation is obtained via approximation of the semidefinite cones by judiciously chosen linear bases, coupled with “delayed constraint generation”. We apply the two convex conic formulations, namely, the semidefinite program and the linear program, to solve the bandgap optimization problems. By comparing the two methods, we demonstrate the efficacy and efficiency of the LP-based algorithm in solving the category of eigenvalue bandgap optimization problems.

The design of metamaterials is currently a fertile research domain. However, most of the metamaterial designs described in the literature arise from physical intuition, and often assume infinite periodicity. There is therefore a need for a design methodology capable of computing patterns and designs involving two different materials where the underlying design variables correspond to a finite set of pixels in a 2-dimensional mesh, and where the goal is a design with prescribed metamaterial properties. This naturally leads to the consideration of binary optimization models in contrast to classical (continuous) gradient-based methods which generically provide continuous solutions that then need to be “rounded” to binary values. While the potential drawback of binary optimization is that its computational complexity is usually NP-hard and hence theoretically unattractive, we show herein that binary optimization combined with a reduced basis approach can relatively efficiently produce very good solutions to metamaterial design problems of interest.

The design of materials is currently a fertile research domain. However, most of the material designs described in the literature arise from physical intuition, and often assume infinite periodicity. There is a need for a design methodology capable of computing patterns and designs involving two different materials where the underlying design variables correspond to a finite set of pixels in a 2-dimensional mesh, and where the goal is a design with prescribed material properties. This naturally leads to the consideration of binary optimization models in contrast to classical (continuous) gradient-based methods, which generically provide continuous solutions that then need to be "rounded" to binary values. While the potential drawback of binary optimization is that its computational complexity is usually NP-hard, and hence theoretically unattractive, we show herein that binary optimization combined with a reduced basis approach can relatively efficiently produce good solutions to material design problems of interest.

In this paper we study the homogeneous conic system F: Ax = 0, x ∈ C∖{0} . We choose a point s̅∈ intC^* that serves as a normalizer and consider computational properties of the normalized system F_s̅ : Ax = 0, s̅^T x = 1, x ∈ C . We show that the computational complexity of solving F via an interior-point method depends only on the complexity value ϑ of the barrier for C and on the symmetry of the origin in the image set H_s̅ := {Ax s̅^Tx = 1, x ∈ C} , where the symmetry of 0 in H_s̅ is sym(0, H_s̅) := max{α : y ∈ H_s̅⇒ -α y ∈ H_s̅} . We show that a solution of F can be computed in O(√(ϑ) ln(ϑ/ sym(0, H_s̅)) interior-point iterations. In order to improve the theoretical and practical computation of a solution of F , we next present a general theory for projective re-normalization of the feasible region F_s̅ and the image set H_s̅ and prove the existence of a normalizer s̅ such that sym(0,H_s̅) ≥ 1/m provided that F has an interior solution. We develop a methodology for constructing a normalizer s̅ such that sym(0, H_s̅) ≥ 1/m with high probability, based on sampling on a geometric random walk with associated probabilistic complexity analysis. While such a normalizer is not itself computable in strongly-polynomial-time, the normalizer will yield a conic system that is solvable in O(√(ϑ) ln(mϑ)) iterations, which is strongly-polynomial-time. Finally, we implement this methodology on randomly generated homogeneous linear programming feasibility problems, constructed to be poorly behaved. Our computational results indicate that the projective re-normalization methodology holds the promise to markedly reduce the overall computation time for conic feasibility problems; for instance we observe a 46

Consider the supposedly simple problem of computing a point in a convex set that is conveyed by a separation oracle with no further information (e.g., no domain ball containing or intersecting the set, etc.). The authors' interest in this problem stems from fundamental issues involving the interplay of (i) the computational complexity of computing a point in the set, (ii) the geometry of the set, and (iii) the stability or conditioning of the set under perturbation. Under suitable definitions of these terms, the authors show herein that problem instances with favorable geometry have favorable computational complexity, validating conventional wisdom. The authors also show a converse of this implication by showing that there exist problem instances that require more computational effort to solve in certain families characterized by unfavorable geometry. This in turn leads, under certain assumptions, to a form of equivalence among computational complexity, geometry, and the conditioning of the set. The authors' measures of the geometry, relative to a given reference point, are based on the radius of a certain domain ball whose intersection with the set contains a certain inscribed ball.The geometry of the set is then measured by the radius of the domain ball, the radius of the inscribed ball, and the ratio between these two radii, the latter of which is called the aspect ratio. The aspect ratio arises in the analysis of many algorithms for convex problems, and its importance in convex algorithm analysis has been well-known for several decades. However, the presence in our bound of terms involving only the radius of the domain ball and only the radius of the inscribed ball are a bit counterintuitive; nevertheless, we show that the computational complexity must involve these terms in addition to the aspect ratio, even when the aspect ratio itself is small. This lower-bound complexity analysis relies on simple features of the separation oracle model; if we instead assume that the set is conveyed by a self-concordant barrier function, then it is an open challenge to prove such a complexity lower-bound.

For a conic linear system of the form Ax ∈K, K a convex cone, several condition measures have been extensively studied in the last dozen years. Among these, Renegar’s condition number $${\mathcal{C}}(A)$$ is arguably the most prominent for its relation to data perturbation, error bounds, problem geometry, and computational complexity of algorithms. Nonetheless, $${\mathcal{C}}(A)$$ is a representation-dependent measure which is usually difficult to interpret and may lead to overly conservative bounds of computational complexity and/or geometric quantities associated with the set of feasible solutions. Herein we show that Renegar’s condition number is bounded from above and below by certain purely geometric quantities associated with A and K; furthermore our bounds highlight the role of the singular values of A and their relationship with the condition number. Moreover, by using the notion of conic curvature, we show how Renegar’s condition number can be used to provide both lower and upper bounds on the width of the set of feasible solutions. This complements the literature where only lower bounds have heretofore been developed.

We attempt a broad exploration of properties and connections between the symmetry function of a convex set S $${S \subset\mathbb{R}^n}$$ and other arenas of convexity including convex functions, convex geometry, probability theory on convex sets, and computational complexity. Given a point $${x \in S}$$ , let sym(x,S) denote the symmetry value of x in S: $${{\bf sym}(x,S):= max\{\alpha \ge 0 : x+\alpha(x-y) \in S {\rm for every} y \in S\}}$$ , which essentially measures how symmetric S is about the point x, and define $${\bf sym}({\it S}):= \max_{x\in S} \, {\bf sym}({\it x,S})$$ x * is called a symmetry point of S if x * achieves the above maximum. The set S is a symmetric set if sym (S)=1. There are many important properties of symmetric convex sets; herein we explore how these properties extend as a function of sym (S) and/or sym (x,S). By accounting for the role of the symmetry function, we reduce the dependence of many mathematical results on the strong assumption that S is symmetric, and we are able to capture and otherwise quantify many of the ways that the symmetry function influences properties of convex sets and functions. The results in this paper include functional properties of sym (x,S), relations with several convex geometry quantities such as volume, distance, and cross-ratio distance, as well as set approximation results, including a refinement of the Löwner-John rounding theorems, and applications of symmetry to probability theory on convex sets. We provide a characterization of symmetry points x * for general convex sets. Finally, in the polyhedral case, we show how to efficiently compute sym(S) and a symmetry point x * using linear programming. The paper also contains discussions of open questions as well as unproved conjectures regarding the symmetry function and its connection to other areas of convexity theory.

We study four measures of problem instance behavior that might account for the observed differences in interior-point method (IPM) iterations when these methods are used to solve semidefinite programming (SDP) problem instances: (i) an aggregate geometry measure related to the primal and dual feasible regions (aspect ratios) and norms of the optimal solutions, (ii) the (Renegar-) condition measure C(d) of the data instance, (iii) a measure of the near-absence of strict complementarity of the optimal solution, and (iv) the level of degeneracy of the optimal solution. We compute these measures for the SDPLIB suite problem instances and measure the sample correlation (CORR) between these measures and IPM iteration counts (solved using the software SDPT3) when these measures have finite values. Our conclusions are roughly as follows: the aggregate geometry measure is highly correlated with IPM iterations (CORR = 0.901), and provides a very good explanation of IPM iterations, particularly for problem instances with solutions of small norm and aspect ratio. The condition measure C(d) is also correlated with IPM iterations, but less so than the aggregate geometry measure (CORR = 0.630). The near-absence of strict complementarity is weakly correlated with IPM iterations (CORR = 0.423). The level of degeneracy of the optimal solution is essentially uncorrelated with IPM iterations.

In this paper we present a general theory for transforming a normalized homogeneous conic system F : Ax = 0, s̄T x = 1, x ∈ C to an equivalent system via projective transformation induced by the choice of a point v̂ in the set H ◦̄ s = {v : s̄−AT v ∈ C∗}. Such a projective transformation serves to pre-condition the conic system into a system that has both geometric and computational properties with certain guarantees. We characterize both the geometric behavior and the computational behavior of the transformed system as a function of the symmetry of v̂ in H ◦̄ s as well as the complexity parameter θ of the barrier for C. Under the assumption that F has an interior solution, H ◦̄ s must contain a point v whose symmetry is at least 1/m; if we can find a point whose symmetry is Ω(1/m) then we can projectively transform the conic system to one whose geometric properties and computational complexity will be strongly-polynomial-time in m and θ. We present a method for generating such a point v̂ based on sampling and on a geometric random walk on H ◦̄ s with associated complexity and probabilistic analysis. Finally, we implement this methodology on randomly generated homogeneous linear programming feasibility problems, constructed to be poorly behaved. Our computational results indicate that the projective pre-conditioning methodology holds the promise to markedly reduce the overall computation time for conic feasibility problems; for instance we observe a 46% decrease in average IPM iterations for 100 randomly generated poorly-behaved problem instances of dimension 1000 × 5000.