Citation Context

...Zhang, 1993), and many learning tasks, where e.g. boosting (Adaboost), support vector machines (Gärtner & Jaggi, 2009; Clarkson, 2010; Ouyang & Gray, 2010), and density estimation (Li & Barron, 2000; =-=Bach et al., 2012-=-) turn out to be such problem instances. Clearly, everyRevisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization X Optimization Domain Complexity of one Frank-Wolfe Iteration Atoms A D = con...

Citation Context

...finite cone this means that the solution has low rank). Due to these two proprieties, conditional gradient methods have attracted much attention in the machine learning community in recent years, see =-=[19, 22, 18, 6, 1, 30, 23, 2]-=-. It is known that in general the convergence rate 1/t is also optimal for this method without further assumptions, as shown in [5, 12, 17]. In case the objective function is both smooth and strongly ...

Citation Context

...nterestingly, the weights wbq do not need to sum to 1, and are not even necessarily positive. 3.1.1 Non-normalized and Negative Weights When weighting samples, it is often assumed, or enforced (as in =-=Bach et al., 2012-=-; Song et al., 2008), that the weights w form a probability distribution. However, there is no technical reason for this requirement, and in fact, the optimal weights do not have this property. Figure...

Citation Context

... = 0 and z0 = µh,p suggested. Either way, it is always the case that zt = z0 + t(µh,p − µh,p,St) where St = {w1, . . . ,wt}. Chen et al. (2010) showed that under some additional assumptions, the herding algorithm, when applied to a RKHS H, greedily minimizes ‖µh,p − µh,p,St‖ 2 H, which, recall, is equal to Dh,p(St). Thus, under certain assumptions, herding and (15) are equivalent. Chen et al. (2010) also showed that under certain restrictions on the RKHS, herding will reduce the discrepancy in a ratio ofO(1/t). However, it is unclear whether those restrictions hold for PWb and p(·). Indeed, Bach et al. (2012) recently shown that these restrictions never hold for infinite-dimensional RKHS, as long as the domain is compact. This result does not immediately apply to our case since Rd is not compact. 5.3 Weighted Sequences Classically, Monte-Carlo and Quasi-Monte Carlo approximations of integrals are unweighted, or more precisely, have a uniform weights. However, it is quite plausible to weight the approximations, i.e. approximate Id[f ] = ∫ [0,1]d f(x)dx using IS,Ξ[f ] = n∑ i=1 ξif(wi) , (16) where Ξ = {ξ1, . . . , ξs} ⊂ R is a set of weights. This lead to the feature map ΨS(x) = [√ ξ1e −ixTw1 . . ....

Citation Context

... f ′(xk−1). 2) Call the linear optimization (LO) oracle to compute yk ∈ Argminx∈X〈pk, x〉. (1.3) 3) Set xk = (1− αk)xk−1 + αkyk for some αk ∈ [0, 1]. In addition to the computation of first-order information, each iteration of the CndG method requires only the solution of a linear optimization subproblem (1.3), while most other first-order methods require the projection over X. Since in some cases it is computationally cheaper to solve (1.3) than to perform projection over X, the CndG method has gained much interests recently from both the machine learning and optimization community (see, e.g.,[1, 2, 3, 7, 6, 11, 15, 14, 16, 17, 18, 22, 27, 28]. In particular, much recent research effort has been devoted to the complexity analysis of the CndG method. For example, it has been shown that if αk in step 3) of the CndG method are properly chosen, then this algorithm can find an -solution of (1.1) (i.e., a point x ∈ X s.t. f(x) − f∗ ≤ ) in at most O(1/) iterations. In fact, such a complexity result has been established for the CndG method under a stronger termination criterion based on the first-order optimality condition of (1.1) (see [17, 18, 11, 14]). Observe that the aforementioned O(1/) bound on gradient evaluations is signific...

Citation Context

...titute of Technology Cambridge, MA 02139 jhuggins@mit.edu Introduction. Recent work has shown many connections between conditional gradient and other first-order optimization methods, such as herding =-=[3]-=- and subgradient descent [2]. By considering a type of proximal conditional method, which we call boosted mirror descent (BMD), we are able to unify all of these algorithms into a single framework, wh...

Citation Context

...ints, and the recovered rates of n−0.49 and n−1 for the i.i.d. and Sobol sequences accord with expectedO(1/ √ n) andO(log(n)/n) bounds from the literature [28, 26]. As witnessed also in other metrics =-=[29]-=-, the herding rate of n−0.96 outpaces its best known bound of dH(Qn, P ) = O(1/ √ n), suggesting an opportunity for sharper analysis. 6 Discussion of Related Work We have developed a quality measure s...

Citation Context

... f 〉 ∣∣Ep(x)f(x)− Ep̂(x)f(x)∣∣ 6 ‖µ− µ̂‖‖f‖ with µ̂ = 1n∑ni=1Φ(xi) Interpretation as conditional gradient • Marginal polytopeM = hull({Φ(x), x ∈ X}) µ • Herding is equivalent to conditional gradient (=-=Bach et al., 2012-=-) min z∈M 1 2 ‖z − µ‖2 – xt+1 = argmaxx∈X 〈Φ(x), µ− zt〉 is the pseudo-sample – Trivial optimization problem... – Three strategies (ρt = 1/(t+ 1), line search, fully corrective) Convergence rates • No ...