Results 1  10
of
26
An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems
, 2009
"... ..."
(Show Context)
Templates for Convex Cone Problems with Applications to Sparse Signal Recovery
, 2010
"... This paper develops a general framework for solving a variety of convex cone problems that frequently arise in signal processing, machine learning, statistics, and other fields. The approach works as follows: first, determine a conic formulation of the problem; second, determine its dual; third, app ..."
Abstract

Cited by 122 (6 self)
 Add to MetaCart
This paper develops a general framework for solving a variety of convex cone problems that frequently arise in signal processing, machine learning, statistics, and other fields. The approach works as follows: first, determine a conic formulation of the problem; second, determine its dual; third, apply smoothing; and fourth, solve using an optimal firstorder method. A merit of this approach is its flexibility: for example, all compressed sensing problems can be solved via this approach. These include models with objective functionals such as the totalvariation norm, ‖W x‖1 where W is arbitrary, or a combination thereof. In addition, the paper also introduces a number of technical contributions such as a novel continuation scheme, a novel approach for controlling the step size, and some new results showing that the smooth and unsmoothed problems are sometimes formally equivalent. Combined with our framework, these lead to novel, stable and computationally efficient algorithms. For instance, our general implementation is competitive with stateoftheart methods for solving intensively studied problems such as the LASSO. Further, numerical experiments show that one can solve the Dantzig selector problem, for which no efficient largescale solvers exist, in a few hundred iterations. Finally, the paper is accompanied with a software release. This software is not a single, monolithic solver; rather, it is a suite of programs and routines designed to serve as building blocks for constructing complete algorithms. Keywords. Optimal firstorder methods, Nesterov’s accelerated descent algorithms, proximal algorithms, conic duality, smoothing by conjugation, the Dantzig selector, the LASSO, nuclearnorm minimization.
Analysis and generalizations of the linearized Bregman method
 SIAM J. IMAGING SCI
, 2010
"... This paper analyzes and improves the linearized Bregman method for solving the basis pursuit and related sparse optimization problems. The analysis shows that the linearized Bregman method has the exact regularization property; namely, it converges to an exact solution of the basis pursuit problem ..."
Abstract

Cited by 36 (9 self)
 Add to MetaCart
This paper analyzes and improves the linearized Bregman method for solving the basis pursuit and related sparse optimization problems. The analysis shows that the linearized Bregman method has the exact regularization property; namely, it converges to an exact solution of the basis pursuit problem whenever its smooth parameter α is greater than a certain value. The analysis is based on showing that the linearized Bregman algorithm is equivalent to gradient descent applied to a certain dual formulation. This result motivates generalizations of the algorithm enabling the use of gradientbased optimization techniques such as line search, Barzilai–Borwein, limited memory BFGS (LBFGS), nonlinear conjugate gradient, and Nesterov’s methods. In the numerical simulations, the two proposed implementations, one using Barzilai–Borwein steps with nonmonotone line search and the other using LBFGS, gave more accurate solutions in much shorter times than the basic implementation of the linearized Bregman method with a socalled kicking technique.
DADMM: A communicationefficient distributed algorithm for separable optimization
 IEEE Trans. Sig. Proc
, 2013
"... ar ..."
(Show Context)
Augmented ℓ1 and nuclearnorm models with a globally linearly convergent algorithm. Rice University CAAM
, 2012
"... This paper studies the longexisting idea of adding a nice smooth function to “smooth ” a nondifferentiable objective function in the context of sparse optimization, in particular, the minimization of ‖x‖1 + 1 2α ‖x‖22, where x is a vector, as well as the minimization of ‖X‖ ∗ + 1 2α ‖X‖2F, where X ..."
Abstract

Cited by 8 (5 self)
 Add to MetaCart
(Show Context)
This paper studies the longexisting idea of adding a nice smooth function to “smooth ” a nondifferentiable objective function in the context of sparse optimization, in particular, the minimization of ‖x‖1 + 1 2α ‖x‖22, where x is a vector, as well as the minimization of ‖X‖ ∗ + 1 2α ‖X‖2F, where X is a matrix and ‖X‖ ∗ and ‖X‖F are the nuclear and Frobenius norms of X, respectively. We show that they let sparse vectors and lowrank matrices be efficiently recovered. In particular, they enjoy exact and stable recovery guarantees similar to those known for the minimization of ‖x‖1 and ‖X‖ ∗ under the conditions on the sensing operator such as its nullspace property, restricted isometry property, spherical section property, or “RIPless ” property. To recover a (nearly) sparse vector x 0, minimizing ‖x‖1+ 1 2α ‖x‖22 returns (nearly) the same solution as minimizing ‖x‖1 whenever α ≥ 10‖x 0 ‖∞. The same relation also holds between minimizing ‖X‖ ∗ + 1 2α ‖X‖2F and minimizing ‖X‖ ∗ for recovering a (nearly) lowrank matrix X 0 if α ≥ 10‖X 0 ‖2. Furthermore, we show that the linearized Bregman algorithm, as well as its two fast variants, for minimizing ‖x‖1 + 1 2α ‖x‖2 2 subject to Ax = b enjoys global linear convergence as long as a nonzero solution exists, and we give an explicit rate of convergence. The convergence property does not require a sparse solution or any properties on A. To our knowledge, this is the best known global convergence result for firstorder sparse optimization algorithms. 1
Coupling forwardbackward with penalty schemes and parallel splitting for constrained variational inequalities
 SIAM Journal on Optimization
"... Abstract. We are concerned with the study of a class of forwardbackward penalty schemes for solving variational inequalities 0 ∈ Ax+NC(x) where H is a real Hilbert space, A: H ⇉ H is a maximal monotone operator, and NC is the outward normal cone to a closed convex set C ⊂ H. Let Ψ: H → R be a conve ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
(Show Context)
Abstract. We are concerned with the study of a class of forwardbackward penalty schemes for solving variational inequalities 0 ∈ Ax+NC(x) where H is a real Hilbert space, A: H ⇉ H is a maximal monotone operator, and NC is the outward normal cone to a closed convex set C ⊂ H. Let Ψ: H → R be a convex differentiable function whose gradient is Lipschitz continuous, and which acts as a penalization function with respect to the constraint x ∈ C. Given a sequence (βn) of penalization parameters which tends to infinity, and a sequence of positive time steps (λn) ∈ ℓ 2 \ ℓ1, we consider the diagonal forwardbackward algorithm xn+1 = (I + λnA) −1(xn − λnβn∇Ψ(xn)). Assuming that (βn) satisfies the growth condition lim sup n→∞ λnβn < 2/θ (where θ is the Lipschitz constant of ∇Ψ), we obtain weak ergodic convergence of the sequence (xn) to an equilibrium for a general maximal monotone operator A. We also obtain weak convergence of the whole sequence (xn) when A is the subdifferential of a proper lowersemicontinuous convex function. As a key ingredient of our analysis, we use the cocoerciveness of the operator ∇Ψ. When specializing our results to coupled systems, we bring new light on Passty’s Theorem, and obtain convergence results of new parallel splitting algorithms for variational inequalities involving coupling in the constraint. We also establish robustness and stability results that account for numerical approximation errors. An illustration to compressive sensing is given. Key words: cocoercive operators; compressive sensing; constrained convex optimization; coupled systems; forwardbackward algorithms; hierarchical optimization; maximal monotone operators; parallel splitting methods; Passty’s Theorem; penalization methods; variational inequalities.
Further results on stable recovery of sparse overcomplete representations in the presence of noise
, 2009
"... Sparse overcomplete representations have attracted much interest recently for their applications to signal processing. In a recent work, Donoho, Elad, and Temlyakov [12] showed that, assuming sufficient sparsity of the ideal underlying signal and approximate orthogonality of the overcomplete dictio ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
Sparse overcomplete representations have attracted much interest recently for their applications to signal processing. In a recent work, Donoho, Elad, and Temlyakov [12] showed that, assuming sufficient sparsity of the ideal underlying signal and approximate orthogonality of the overcomplete dictionary, the sparsest representation can be found, at least approximately if not exactly, by either an orthogonal greedy algorithm or ¢ ¡ bynorm minimization subject to a noise tolerance constraint. In this paper, we sharpen the approximation bounds under more relaxed conditions. We also derive analogous results for a stepwise projection algorithm.
Lexicographically Optimal Routing for Wireless Sensor Networks with Multiple Sinks
, 2008
"... In wireless sensor networks (WSNs), the field information (e.g., temperature, humidity, airflow) is acquired via several batteryequipped wireless sensors and is relayed towards a sink node. As the size of the WSNs increases, it becomes inefficient (in terms of power consumption) to gather all info ..."
Abstract

Cited by 6 (0 self)
 Add to MetaCart
In wireless sensor networks (WSNs), the field information (e.g., temperature, humidity, airflow) is acquired via several batteryequipped wireless sensors and is relayed towards a sink node. As the size of the WSNs increases, it becomes inefficient (in terms of power consumption) to gather all information in a single sink. To tackle this problem, one can increase the number of sinks. The set of sensor nodes sending data to sink k is called commodity k. In this paper, we formulate the lexicographically optimal commodity lifetime routing problem. A stepwise centralized algorithm, called the lexicographically optimal commodity lifetime (LOCL) algorithm, is proposed which can obtain the optimal routing solution and lead to lexicographical fairness among commodity lifetimes. We then show that under certain assumptions, the lexicographical optimality among commodity lifetimes can be achieved by providing lexicographical optimality among node lifetimes. This motivates us to propose our second algorithm, called the lexicographically optimal node lifetime (LONL) algorithm, which suitable for practical implementation. Simulation results show that our proposed LOCL and LONL algorithms increase the normalized commodity and node lifetimes compared to the maximum lifetime with multiple sinks (MLMS) [1] and lexicographical maxmin fair (LMM) [2] routing algorithms.
Convex perturbations for scalable semidefinite programming
 In International Conference on Artificial Intelligence and Statistics (AISTATS
, 2009
"... Abstract Many important machine learning problems are modeled and solved via semidefinite programs; examples include metric learning, nonlinear embedding, and certain clustering problems. Often, offtheshelf software is invoked for the associated optimization, which can be inappropriate due to exc ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
(Show Context)
Abstract Many important machine learning problems are modeled and solved via semidefinite programs; examples include metric learning, nonlinear embedding, and certain clustering problems. Often, offtheshelf software is invoked for the associated optimization, which can be inappropriate due to excessive computational and storage requirements. In this paper, we introduce the use of convex perturbations for solving semidefinite programs (SDPs), and for a specific perturbation we derive an algorithm that has several advantages over existing techniques: a) it is simple, requiring only a few lines of MATLAB, b) it is a firstorder method, and thereby scalable, and c) it can easily exploit the structure of a given SDP (e.g., when the constraint matrices are lowrank, a situation common to several machine learning SDPs). A pleasant byproduct of our method is a fast, kernelized version of the largemargin nearest neighbor metric learning algorithm