Citation Context

...Zhang, 1993), and many learning tasks, where e.g. boosting (Adaboost), support vector machines (Gärtner & Jaggi, 2009; Clarkson, 2010; Ouyang & Gray, 2010), and density estimation (Li & Barron, 2000; =-=Bach et al., 2012-=-) turn out to be such problem instances. Clearly, everyRevisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization X Optimization Domain Complexity of one Frank-Wolfe Iteration Atoms A D = con...

Citation Context

...finite cone this means that the solution has low rank). Due to these two proprieties, conditional gradient methods have attracted much attention in the machine learning community in recent years, see =-=[19, 22, 18, 6, 1, 30, 23, 2]-=-. It is known that in general the convergence rate 1/t is also optimal for this method without further assumptions, as shown in [5, 12, 17]. In case the objective function is both smooth and strongly ...

Citation Context

...nterestingly, the weights wbq do not need to sum to 1, and are not even necessarily positive. 3.1.1 Non-normalized and Negative Weights When weighting samples, it is often assumed, or enforced (as in =-=Bach et al., 2012-=-; Song et al., 2008), that the weights w form a probability distribution. However, there is no technical reason for this requirement, and in fact, the optimal weights do not have this property. Figure...

Citation Context

...titute of Technology Cambridge, MA 02139 jhuggins@mit.edu Introduction. Recent work has shown many connections between conditional gradient and other first-order optimization methods, such as herding =-=[3]-=- and subgradient descent [2]. By considering a type of proximal conditional method, which we call boosted mirror descent (BMD), we are able to unify all of these algorithms into a single framework, wh...

Citation Context

... f 〉 ∣∣Ep(x)f(x)− Ep̂(x)f(x)∣∣ 6 ‖µ− µ̂‖‖f‖ with µ̂ = 1n∑ni=1Φ(xi) Interpretation as conditional gradient • Marginal polytopeM = hull({Φ(x), x ∈ X}) µ • Herding is equivalent to conditional gradient (=-=Bach et al., 2012-=-) min z∈M 1 2 ‖z − µ‖2 – xt+1 = argmaxx∈X 〈Φ(x), µ− zt〉 is the pseudo-sample – Trivial optimization problem... – Three strategies (ρt = 1/(t+ 1), line search, fully corrective) Convergence rates • No ...

Citation Context

... z∈Z 〈 φ(z), η− 1 n n−1∑ i=1 φ(z̄i) 〉 H, for n ≥ 2. Herding can be understood as an iterative greedy optimization procedure for finding examples z̄1, . . . , z̄n that minimize ‖η − 1n ∑n i=1 φ(z̄i)‖H =-=[2]-=-. This interpretation shows that the target vector, η, is not restricted to be an embedded distribution, so herding can be applied to arbitrary vectors in H. Doing so for µ̃T+1 yields a set S̄T+1 = {z...

Citation Context

...ints, and the recovered rates of n−0.49 and n−1 for the i.i.d. and Sobol sequences accord with expectedO(1/ √ n) andO(log(n)/n) bounds from the literature [28, 26]. As witnessed also in other metrics =-=[29]-=-, the herding rate of n−0.96 outpaces its best known bound of dH(Qn, P ) = O(1/ √ n), suggesting an opportunity for sharper analysis. 6 Discussion of Related Work We have developed a quality measure s...

Citation Context

...aoui (Inria Grenoble, LEAR team). 6.4. On the Equivalence between Herding and Conditional Gradient Algorithms Participants: Francis Bach [correspondant], Simon Lacoste-Julien, Guillaume Obozinski. In =-=[5]-=-, we show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm–namely a conditional gradient algorithm minimizing a quadratic moment discrepa...