Citation Context

...ap [18, Section 5.1]. The computational cost of this additional step is equivalent to one pass of the dataset, thus it does not affect the overall complexity. 4.2 Other related work Another way to approach problem (1) is to reformulate it as a constrained optimization problem minimize 1 n n∑ i=1 φi(zi) + g(x) (26) subject to aTi x = zi, , i = 1, . . . , n, 13 and solve it by ADMM type of operator-splitting methods (e.g., [19]). In fact, as shown in [8], the batch primal-dual algorithm (21)-(23) is equivalent to a pre-conditioned ADMM (or inexact Uzawa method; see, e.g., [47]). Several authors [42, 28, 37, 49] have considered a more general formulation than (26), where each φi is a function of the whole vector z ∈ Rn. They proposed online or stochastic versions of ADMM which operate on only one φi in each iteration, and obtained sublinear convergence rates. However, their cost per iteration is O(nd) instead of O(d). Suzuki [38] considered a problem similar to (1), but with more complex regularization function g, meaning that g does not have a simple proximal mapping. Thus primal updates such as step (5) or (9) in SPDC and similar steps in SDCA cannot be computed efficiently. He proposed an algorith...

Citation Context

... algorithm (HONOR) which combines the quasi-Newton method and the gradient descent method (Gong and Ye, 2015). However, the MS algorithm does not admit a closed-form solution for graph-guided regularized optimization problems and hence leads to an expensive per-iteration computational cost. When A or B is non-diagonal, neither the SCP algorithm nor the GIST algorithm is efficient for solving problem (1) since the proximal mapping of r(·) is typically not available. Another related stream of works are the ADMM-type algorithms which are suitable to solve problem (1) when A or B is not diagonal (Zhong and Kwok, 2013; Zhang and Kwok, 2014; Wang et al., 2014; Zhao et al., 2015). Such a kind of algorithms have recently been shown effective to handle some nonconvex optimization problems (Magnsson et al., 2014; Jiang et al., 2014; Hong et al., 2015; Yang et al., 2015; Wang et al., 2015a,b; Li and Pong, 2015). However, the results of (Magnsson et al., 2014; Jiang et al., 2014) require a not well-justified assumption about the generated iterates, while some other works focus on certain specific problems such as the consensus and sharing problems (Hong et al., 2015) and the background/foreground extraction probl...

Citation Context

...large, the ADMM algorithm suffers from the problem of high memory demand for solving (4). Fortunately, the marriage of the recently developed stochastic average (SA) algorithms and ADMM, i.e., SA-ADMM=-=[34]-=-, can be utilized to optimize (4). Different from standard ADMM, SA-ADMM adopts the linearization technique which can be deployed to avoid the computation of matrix inversion in our case, and utilizes...