In this paper, we study a general nonsmooth, nonconvex optimization problem over a cross product of spheres, suitable for various well-known variants of the canonical correlation analysis (CCA) and singular value decomposition (SVD) problems that promote sparsity and smoothness. We propose an alternating minimization algorithm for a smooth approximation of this problem and study its rate of convergence. Numerical experiments demonstrate the potential of the suggested method.

We consider the support vector data description problem which, given a set of data points, seeks to find a ball that minimizes an objective function incorporating both the radius of the ball and a penalty for any data point located outside the ball. We present a reduction of the problem to an unconstrained minimization of a strongly convex function, enabling it to be solved by a dual-based accelerated gradient method.

We consider the class of convex composite minimization problems which consists of minimizing the sum of two nonsmooth extended valued convex functions, with one which is composed with a linear map. Convergence rate guarantees for first order methods on this class of problems often require the additional assumption of Lipschitz continuity of the nonsmooth objective function composed with the linear map. We introduce a theoretical framework where the restrictive Lipschitz continuity of this function is not required. Building on a novel dual representation of the so-called Pasch-Hausdorff envelope, we derive an exact Lipshitz regularization for this class of problems. We then show how the aforementioned result can be utilized in establishing function values-based rates of convergence in terms of the original data. Throughout, we provide examples and applications which illustrate the potential benefits of our approach.

Fisher linear discriminant analysis (FLDA or LDA) is a well-known technique for dimension reduction and classification. The method was first formulated in 1936 by Fisher in the one-dimensional setting. In this paper, we will examine the LDA problem using a different objective function. Instead of maximizing the sum of all distances between all classes, we will define an objective function that will maximize the minimum separation among all distances between all classes. This leads to a difficult nonconvex optimization problem. We present a branch and bound method for the problem in the case where the reduction is to the one-dimensional space.

We consider the problem of minimizing the sum of a smooth nonconvex function and a nonsmooth convex function over a compact embedded submanifold. We describe an algorithm, which we refer to as "dynamic smoothing gradient descent on manifolds" (DSGM), that is based on applying Riemmanian gradient steps on a series of smooth approximations of the objective function that are determined by a diminishing sequence of smoothing parameters. The DSGM algorithm is simple and can be easily employed for a broad class of problems without any complex adjustments. We show that all accumulation points of the sequence generated by the method are stationary. We devise a convergence rate of O(1/k(1/3)) in terms of an optimality measure that can be easily computed. Numerical experiments illustrate the potential of the DSGM method.

Fisher linear discriminant analysis is a well-known technique for dimensionality reduction and classification. The method was first formulated in 1936 by Fisher. In this paper we concentrate on three different formulations of the multi-dimensional problem. We provide a mathematical explanation why two of the formulations are equivalent and prove that this equivalency can be extended to a broader class of objective functions. The second contribution is a rate of convergence of a fixed point method for solving the third model.

This paper develops and studies a feasible directions approach for the minimization of a continuous function over linear constraints in which the update directions belong to a predetermined finite set spanning the feasible set. These directions are recurrently investigated in a cyclic semi-random order, where the stepsize of the update is determined via univariate optimization. We establish that any accumulation point of this optimization procedure is a stationary point of the problem, meaning that the directional derivative in any feasible direction is nonnegative. To assess and establish a rate of convergence, we develop a new optimality measure that acts as a proxy for the stationarity condition, and substantiate its role by showing that it is coherent with first-order conditions in specific scenarios. Finally we prove that our method enjoys a sublinear rate of convergence of this optimality measure in expectation.

The paper considers the sparse envelope function, defined as the biconjugate of the sum of a squared ℓ 2 -norm function and the indicator of the set of k -sparse vectors. It is shown that both function and proximal values of the sparse envelope function can be reduced into a one-dimensional search that can be efficiently performed in linear time complexity in expectation. The sparse envelope function naturally serves as a regularizer that can handle both sparsity and grouping information in inverse problems, and can also be utilized in sparse support vector machine problems.

We consider a nonconvex optimization problem consisting of maximizing the difference of two convex functions. We present a randomized method that requires low computational effort at each iteration. The described method is a randomized coordinate descent method employed on the so-called Toland-dual problem. We prove subsequence convergence to dual stationarity points, a new notion that we introduce and which is shown to be tighter than standard criticality. An almost sure rate of convergence of an optimality measure of the dual sequence is proven. We demonstrate the potential of our results on three principal component analysis models resulting in extremely simple algorithms.

Coordinate descent algorithms are popular in machine learning and large-scale data analysis problems due to their low computational cost iterative schemes and their improved performances. In this work, we define a monotone accelerated coordinate gradient descent-type method for problems consisting of minimizing $$f+g$$ , where f is quadratic and g is nonsmooth and non-separable and has a low-complexity proximal mapping. The algorithm is enabled by employing the forward–backward envelope, a composite envelope that possess an exact smooth reformulation of $$f+g$$ . We prove the algorithm achieves a convergence rate of $$O(1/k^{1.5})$$ in terms of the original objective function, improving current coordinate descent-type algorithms. In addition, we describe an adaptive variant of the algorithm that backtracks the spectral information and coordinate Lipschitz constants of the problem. We numerically examine our algorithms on various settings, including two-dimensional total-variation-based image inpainting problems, showing a clear advantage in performance over current coordinate descent-type methods.

This paper studies the class of nonsmooth nonconvex problems in which the difference between a continuously differentiable function and a convex nonsmooth function is minimized over linear constraints. Our goal is to attain a point satisfying the stationarity necessary optimality condition, defined as the lack of feasible descent directions. Although elementary in smooth optimization, this condition is nontrivial when the objective function is nonsmooth, and, correspondingly, there are very few methods that obtain stationary points in such settings. We prove that stationarity in our model can be characterized by a finite number of directions and develop two methods, one deterministic and one random, that use these directions to obtain stationary points. Numerical experiments illustrate the benefit of obtaining a stationary point and the advantage of using the random method to do so.

This paper studies a general form problem in which a lower bounded continuously differentiable function is minimized over a block separable set incorporating a group sparsity expression as a constraint or a penalty (or both) in the group sparsity setting. This class of problems is generally hard to solve, yet highly applicable in numerous practical settings. Particularly, we study the proximal mapping that includes group-sparsity terms, and derive an efficient method to compute it. Necessary optimality conditions for the problem are devised, and a hierarchy between stationary-based and coordinate-wised based conditions is established. Methods that obtain points satisfying the optimality conditions are presented, analyzed and tested in applications from the fields of investment and graph theory.

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting from unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.

This paper presents the FOM MATLAB toolbox for solving convex optimization problems using first-order methods. The diverse features of the eight solvers included in the package are illustrated through a collection of examples of different nature.

We suggest a majorization-minimization method for solving nonconvex minimization problems. The method is based on minimizing at each iterate a properly constructed consistent majorizer of the objective function. We describe a variety of classes of functions for which such a construction is possible. We introduce an inexact variant of the method, in which only approximate minimization of the consistent majorizer is performed at each iteration. Both the exact and the inexact algorithms are shown to be descent methods whose accumulation points have a property which is stronger than standard stationarity. We give examples of cases in which the exact method can be applied. Finally, we show that the inexact method can be applied to a specific problem, called sparse source localization, by utilizing a fast optimization method on a smooth convex dual of its subproblems.

This paper studies a class of problems consisting of minimizing a continuously differentiable function penalized with the so-called $\ell_0$-norm over a symmetric set. These problems are hard to solve, yet prominent in many fields and applications. We first study the proximal mapping with respect to the $\ell_0$-norm over symmetric sets, and provide an efficient method to attain it. The method is then improved for symmetric sets satisfying a sub-modularity-like property, which we call second order monotonicity (SOM). It is shown that many important symmetric sets, such as the $\ell_1,\ell_2, \ell_{\infty}$-balls, the simplex and the full-simplex, satisfy this SOM property. We then develop, under the validity of the SOM property, necessary optimality conditions, and corresponding algorithms that are guaranteed to converge to points satisfying the aforementioned optimality conditions. We prove the existence of a hierarchy between the optimality conditions, and consequently between the corresponding algorithms.

Alternating minimization, or Fienup methods, have a long history in phase retrieval. We provide new insights related to the empirical and theoretical analysis of these algorithms when used with Fourier measurements and combined with convex priors. In particular, we show that Fienup methods can be viewed as performing alternating minimization on a regularized nonconvex least-squares problem with respect to amplitude measurements. Furthermore, we prove that under mild additional structural assumptions on the prior (semialgebraicity), the sequence of signal estimates has a smooth convergent behavior toward a critical point of the nonconvex regularized least-squares objective. Finally, we propose an extension to Fienup techniques, based on a projected gradient descent interpretation and acceleration using inertial terms. We demonstrate experimentally that this modification combined with an l(1) prior constitutes a competitive approach for sparse phase retrieval.

We introduce the notion of predicted decrease approximation (PDA) for constrained convex optimization, a flexible framework which includes as special cases known algorithms such as generalized conditional gradient, proximal gradient, greedy coordinate descent for separable constraints and working set methods for linear equality constraints with bounds. The new scheme allows the development of a unified convergence analysis for these methods. We further consider a partially strongly convex nonsmooth model and show that dual application of PDA-based methods yields new sublinear convergence rate estimates in terms of both primal and dual objectives. As an example of an application, we provide an explicit working set selection rule for SMO-type methods for training the support vector machine with an improved primal convergence analysis.

We devise an algorithm for finding the global optimal solution of the so-called optimal power flow problem for a class of power networks with a tree topology, also called radial networks, for which an efficient and reliable algorithm was not previously known. The algorithm we present is called the tree reduction/expansion method, and is based on an equivalence between the input network and a single-node network. Finally, our numerical experiments demonstrate the reliability and robustness of our algorithm.