Online learning for matrix factorization and sparse coding

Online learning for matrix factorization and sparse coding

Cached

Download Links

by Julien Mairal , Francis Bach , Ecole Normale Supérieure , Guillermo Sapiro

Citations:

BibTeX

@MISC{Mairal_onlinelearning,
    author = {Julien Mairal and Francis Bach and Ecole Normale Supérieure and Guillermo Sapiro},
    title = {Online learning for matrix factorization and sparse coding},
    year = {}
}

Bookmark

OpenURL

Abstract

Sparse coding—that is, modelling data vectors as sparse linear combinations of basis elements—is widely used in machine learning, neuroscience, signal processing, and statistics. This paper focuses on the large-scale matrix factorization problem that consists of learning the basis set, adapting it to specific data. Variations of this problem include dictionary learning in signal processing, non-negative matrix factorization and sparse principal component analysis. In this paper, we propose to address these tasks with a new online optimization algorithm, based on stochastic approximations, which scales up gracefully to large datasets with millions of training samples, and extends naturally to various matrix factorization formulations, making it suitable for a wide range of learning problems. A proof of convergence is presented, along with experiments with natural images and genomic data demonstrating that it leads to state-of-the-art performance in terms of speed and optimization for both small and large datasets.

Citations

1524	Matrix computations - Golub, Loan - 1996
998	Regression shrinkage and selection via the lasso - Tibshirani - 1996
612	A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998
554	Nonlinear Programming. Athena Scientific - Bertsekas - 1995
478	Least angle regression - Efron - 2004
464	Algorithms for non-negative matrix factorization - Lee, Seung - 2001
427	Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research - Olshausen, Field - 1997
283	From few to many: illumination cone models for face recognition under variable lighting and pose - Georghiades, Belhumeur, et al.
238	Y: Model selection and estimation in regression with grouped variables - Yuan, Lin
203	Relations between two sets of variates - Hotelling - 1936
200	Non-negative matrix factorization with sparseness constraints - Hoyer - 2004
188	Learning overcomplete representations - Lewicki, Sejnowski - 2000
181	BRUCKSTEIN A.: K-svd: An algorithm for designing overcomplete dictionaries for sparse representation - AHARON, ELAD - 2006
180	Matrix Differential Calculus with Applications to Statistics and Econometrics - Magnus, Neudecker - 1988
166	A.: Perturbation Analysis of Optimization Problems - Bonnans, Shapiro - 2000
159	Regularization and Variable Selection via the Elastic Net - Zou, Hastie - 2008
149	Blind Source Separation by Sparse Decomposition,” tech - Zibulevsky, Pearlmutter - 1999
138	Image denoising via sparse and redundant representations over learned dictionaries - Elad, Aharon
121	A new approach to variable selection in least squares problems - Osborne, Presnell, et al. - 2000
115	A direct formulation for sparse PCA using semidefinite programming - d’Aspremont, Ghaoui, et al. - 2007
111	Selftaught learning: transfer learning from unlabeled data - Raina, Battle, et al.
110	Algorithms for simultaneous sparse approximation. part I: Greedy pursuit - Tropp, Gilbert, et al.
108	Efficient sparse coding algorithms - Lee, Battle, et al.
100	Stochastic approximation and recursive algorithms and applications - Kushner, Yin - 2003
100	der Vaart. Asymptotic Statistics - van - 1998
92	Pathwise coordinate optimization - Friedman, Hastie, et al.
86	Simultaneous analysis of lasso and dantzig selector. Preprint Submitted to the Annals of Statistics - Bickel, Ritov, et al. - 2007
83	Tibshirani R: Sparse principal component analysis - Zou, Hastie
81	Consistency of the Group Lasso and Multiple Kernel Learning - Bach
79	Convex Analysis and Nonlinear Optimization, Theory and Examples, Second Edition - Borwein, Lewis - 2006
76	Projected gradient methods for nonnegative matrix factorization - Lin - 2007
73	Non-negative sparse coding - Hoyer - 2002
72	Linear Spatial Pyramid Matching using Sparse Coding for Image Classification - Yang, Yu, et al. - 2009
71	Gradient methods for minimizing composite objective function - Nesterov - 2007
69	K: Sparsity and smoothness via the fused LASSO - Tibshirani, Saunders, et al.
68	Sparse solutions to linear inverse problems with multiple measurement vectors - Rao, Kreutz-Delgado - 1998
67	Acquiring linear subspaces for face recognition under variable lighting - Lee, Ho, et al. - 2005
65	The tradeoffs of large scale learning - Bottou, Bousquet - 2008
62	Sparse representation for color image restoration - Mairal, Elad, et al. - 2008
61	Penalized regressions: the bridge versus the Lasso - Fu - 1998
57	Matrix factorization techniques for recommender systems - Koren, Bell, et al.
53	Learning and Example Selection for Object and Pattern Recognition - Sung - 1996
52	A modified principal component technique based on - Jolliffe, Trendafilov, et al. - 2003
49	Supervised dictionary learning - Mairal, Bach, et al. - 2008
47	J.P.: Group lasso with overlap and graph lasso - Jacob, Obozinski, et al. - 2009
46	A.: Non-Local Sparse Models for Image Restoration - Mairal, Bach, et al.
46	Simultaneous variable selection - Turlach, Venables, et al. - 2005
45	Discriminative learned dictionaries for local image analysis - Mairal, Bach, et al. - 2008
45	Structured variable selection with sparsity-inducing norms - Jenatton, Audibert, et al.
45	Joint covariate selection and joint subspace selection for multiple classification problems - Obozinski, Taskar, et al. - 2010