Results 1  10
of
22
Parallel multiblock ADMM with o(1/k) convergence,” Preprint, available online at arXiv: 1312.3040
, 2014
"... Abstract. This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize f1(x1) + · · ·+ fN (xN) subject to A1x1 + · · ·+ANxN = c, x1 ∈ X1,..., xN ∈ XN. The algorithm decomposes the original problem into N ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
(Show Context)
Abstract. This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize f1(x1) + · · ·+ fN (xN) subject to A1x1 + · · ·+ANxN = c, x1 ∈ X1,..., xN ∈ XN. The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobiantype algorithm is well suited for distributed computing and is particularly attractive for solving certain largescale problems. This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic GaussSeidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices Ai are mutually nearorthogonal and have full columnrank. Secondly, for general matrices Ai, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k). In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms. We implemented our algorithm on Amazon EC2, an ondemand public computing cloud, and report its performance on very largescale basis pursuit problems with distributed data. Key words. alternating direction method of multipliers, ADMM, parallel and distributed computing, convergence rate
Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems
"... ar ..."
(Show Context)
Asynchronous Distributed ADMM for Consensus Optimization
"... Distributed optimization algorithms are highly attractive for solving big data problems. In particular, many machine learning problems can be formulated as the global consensus optimization problem, which can then be solved in a distributed manner by the alternating direction method of multiplier ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
Distributed optimization algorithms are highly attractive for solving big data problems. In particular, many machine learning problems can be formulated as the global consensus optimization problem, which can then be solved in a distributed manner by the alternating direction method of multipliers (ADMM) algorithm. However, this suffers from the straggler problem as its updates have to be synchronized. In this paper, we propose an asynchronous ADMM algorithm by using two conditions to control the asynchrony: partial barrier and bounded delay. The proposed algorithm has a simple structure and good convergence guarantees (its convergence rate can be reduced to that of its synchronous counterpart). Experiments on different distributed ADMM applications show that asynchrony reduces the time on network waiting, and achieves faster convergence than its synchronous counterpart in terms of the wall clock time. 1.
A stochastic coordinate descent primaldual algorithm and applications to largescale composite optimization,
, 2014
"... AbstractBased on the idea of randomized coordinate descent of αaveraged operators, a randomized primaldual optimization algorithm is introduced, where a random subset of coordinates is updated at each iteration. The algorithm builds upon a variant of a recent (deterministic) algorithm proposed b ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
(Show Context)
AbstractBased on the idea of randomized coordinate descent of αaveraged operators, a randomized primaldual optimization algorithm is introduced, where a random subset of coordinates is updated at each iteration. The algorithm builds upon a variant of a recent (deterministic) algorithm proposed by Vũ and Condat that includes the well known ADMM as a particular case. The obtained algorithm is used to solve asynchronously a distributed optimization problem. A network of agents, each having a separate cost function containing a differentiable term, seek to find a consensus on the minimum of the aggregate objective. The method yields an algorithm where at each iteration, a random subset of agents wake up, update their local estimates, exchange some data with their neighbors, and go idle. Numerical results demonstrate the attractive performance of the method. The general approach can be naturally adapted to other situations where coordinate descent convex optimization algorithms are used with a random choice of the coordinates.
Stochastic GradientPush for Strongly Convex Functions on TimeVarying Directed Graphs
, 2014
"... We investigate the convergence rate of the recently proposed subgradientpush method for distributed optimization over timevarying directed graphs. The subgradientpush method can be implemented in a distributed way without requiring knowledge of either the number of agents or the graph sequence; ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
We investigate the convergence rate of the recently proposed subgradientpush method for distributed optimization over timevarying directed graphs. The subgradientpush method can be implemented in a distributed way without requiring knowledge of either the number of agents or the graph sequence; each node is only required to know its outdegree at each time. Our main result is a convergence rate of O ((ln t)/t) for strongly convex functions with Lipschitz gradients even if only stochastic gradient samples are available; this is asymptotically faster than the O (ln t)/
Distributed online modified greedy algorithm for networked storage operation under uncertainty
 Online]. Available: http://arxiv.org/pdf/1406.4615v2.pdf
, 2014
"... iv ..."
(Show Context)
Linear time average consensus on fixed graphs and implications for decentralized optimization and multiagent control,” http://arxiv.org/pdf/1411.4186v5.pdf
, 2015
"... We describe a protocol for the average consensus problem on any fixed undirected graph whose convergence time scales linearly in the total number nodes n. More precisely, we provide a protocol which results in each node having a value within an of the initial average after O n ln x(1)−x12 itera ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
We describe a protocol for the average consensus problem on any fixed undirected graph whose convergence time scales linearly in the total number nodes n. More precisely, we provide a protocol which results in each node having a value within an of the initial average after O n ln x(1)−x12 iterations. The protocol is completely distributed, with the exception of requiring each node to know an upper bound U on the total number of nodes which is correct within a constant multiplicative factor. We discuss applications of our nearlylinear protocol to questions in decentralized optimization and multiagent control connected to the consensus problem. In particular, we develop a distributed protocol for minimizing an average of (possibly nondifferentiable) convex functions (1/n) ∑n i=1 fi(θ), in the setting where only node i in an undirected, connected graph knows the function fi(θ). Under the same assumption about all nodes knowing U, and additionally assuming that the subgradients of each fi(θ) have norms upper bounded by some constant L known to the nodes, after T iterations our protocol has error which is O(L n/T).
Convergence Rates of Distributed NesterovLike Gradient Methods on Random Networks
"... Abstract—We consider distributed optimization in random networks where nodes cooperatively minimize the sum of their individual convex costs. Existing literature proposes distributed gradientlike methods that are computationally cheap and resilient to link failures, but have slow convergence rates ..."
Abstract

Cited by 2 (2 self)
 Add to MetaCart
(Show Context)
Abstract—We consider distributed optimization in random networks where nodes cooperatively minimize the sum of their individual convex costs. Existing literature proposes distributed gradientlike methods that are computationally cheap and resilient to link failures, but have slow convergence rates. In this paper, we propose accelerated distributed gradient methods that 1) are resilient to link failures; 2) computationally cheap; and 3) improve convergence rates over other gradient methods. We model the network by a sequence of independent, identically distributed random matrices drawn from the set of symmetric, stochastic matrices with positive diagonals. The network is connected on average and the cost functions are convex, differentiable, with Lipschitz continuous and bounded gradients. We design two distributed Nesterovlike gradient methods that modify the D–NG and D–NC methods that we proposed for static networks. We prove their convergence rates in terms of the expected optimality gap at the cost function. Let and be the number of pernode gradient evaluations and pernode communications, respectively. Then the modified D–NG achieves rates and, and the modified D–NC rates and, where is arbitrarily small. For comparison, the standard distributed gradient method cannot do better than and, on the same class of cost functions (even for static networks). Simulation examples illustrate our analytical findings. Index Terms—Consensus, convergence rate, distributed optimization, Nesterov gradient, random networks. I.
Timeaverage optimization with nonconvex decision set and its convergence,” in CDC,
 Proceedings IEEE,
, 2014
"... AbstractThis paper considers timeaverage optimization, where a decision vector is chosen every time step within a (possibly nonconvex) set, and the goal is to minimize a convex function of the time averages subject to convex constraints on these averages. Such problems have applications in networ ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
AbstractThis paper considers timeaverage optimization, where a decision vector is chosen every time step within a (possibly nonconvex) set, and the goal is to minimize a convex function of the time averages subject to convex constraints on these averages. Such problems have applications in networking and operations research, where decisions can be constrained to discrete sets and time averages can represent bit rates, power expenditures, and so on. These problems can be solved by Lyapunov optimization. This paper shows that a simple driftbased algorithm, related to a classical dual subgradient algorithm, converges to an optimal solution within O(1/ 2 ) time steps. However, when the problem has a unique vector of Lagrange multipliers, the algorithm is shown to have a transient phase and a steady state phase. By restarting the time averages after the transient phase, the total convergence time is improved to O(1/ ) under a locallypolyhedron assumption, and to O(1/ 1.5 ) under a locallysmooth assumption.
Convergence analysis of ADMMbased power system mode estimation under asynchronous widearea communication delays
 in Proc. IEEE PES General Meeting
, 2015
"... Abstract — In our recent paper [1], we proposed a distributed PMUPDC architecture for estimating power system oscillation modes by integrating a Pronybased algorithm with Alternating Direction Method of Multipliers (ADMM). A critical assumption behind the proposed method was that the communicatio ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
(Show Context)
Abstract — In our recent paper [1], we proposed a distributed PMUPDC architecture for estimating power system oscillation modes by integrating a Pronybased algorithm with Alternating Direction Method of Multipliers (ADMM). A critical assumption behind the proposed method was that the communication between local PDCs and the central averager is completely synchronized. In realistic widearea networks, however, such synchronous communication may not always be possible. In this paper we address this issue of asynchronous communication, and its impact on the convergence of the distributed estimation. We first impose a probability model for the communication delays between the central PDC and the local PDCs, and then implement two strategies of averaging at the central PDC based on a chosen delay threshold. We carry out simulations to show possible instabilities and sensitivities of the ADMM convergence on delay distribution parameters under these two averaging strategies. Our results exhibit a broad view of how the convergence of distributed estimation algorithms in physical processes depends strongly on the uncertainties in the underlying communications in a generic cyberphysical system. I.