Results 1  10
of
26
Online learning for matrix factorization and sparse coding
, 2010
"... Sparse coding—that is, modelling data vectors as sparse linear combinations of basis elements—is widely used in machine learning, neuroscience, signal processing, and statistics. This paper focuses on the largescale matrix factorization problem that consists of learning the basis set in order to ad ..."
Abstract

Cited by 317 (31 self)
 Add to MetaCart
(Show Context)
Sparse coding—that is, modelling data vectors as sparse linear combinations of basis elements—is widely used in machine learning, neuroscience, signal processing, and statistics. This paper focuses on the largescale matrix factorization problem that consists of learning the basis set in order to adapt it to specific data. Variations of this problem include dictionary learning in signal processing, nonnegative matrix factorization and sparse principal component analysis. In this paper, we propose to address these tasks with a new online optimization algorithm, based on stochastic approximations, which scales up gracefully to large data sets with millions of training samples, and extends naturally to various matrix factorization formulations, making it suitable for a wide range of learning problems. A proof of convergence is presented, along with experiments with natural images and genomic data demonstrating that it leads to stateoftheart performance in terms of speed and optimization for both small and large data sets.
Dictionaries for Sparse Representation Modeling
"... Sparse and redundant representation modeling of data assumes an ability to describe signals as linear combinations of a few atoms from a prespecified dictionary. As such, the choice of the dictionary that sparsifies the signals is crucial for the success of this model. In general, the choice of a p ..."
Abstract

Cited by 108 (3 self)
 Add to MetaCart
Sparse and redundant representation modeling of data assumes an ability to describe signals as linear combinations of a few atoms from a prespecified dictionary. As such, the choice of the dictionary that sparsifies the signals is crucial for the success of this model. In general, the choice of a proper dictionary can be done using one of two ways: (i) building a sparsifying dictionary based on a mathematical model of the data, or (ii) learning a dictionary to perform best on a training set. In this paper we describe the evolution of these two paradigms. As manifestations of the first approach, we cover topics such as wavelets, wavelet packets, contourlets, and curvelets, all aiming to exploit 1D and 2D mathematical models for constructing effective dictionaries for signals and images. Dictionary learning takes a different route, attaching the dictionary to a set of examples it is supposed to serve. From the seminal work of Field and Olshausen, through the MOD, the KSVD, the Generalized PCA and others, this paper surveys the various options such training has to offer, up to the most recent contributions and structures.
Nonnegative sparse modeling of textures
 In SSVM
, 2007
"... Abstract. This paper presents a statistical model for textures that uses a nonnegative decomposition on a set of local atoms learned from an exemplar. This model is described by the variances and kurtosis of the marginals of the decomposition of patches in the learned dictionary. A fast sampling al ..."
Abstract

Cited by 30 (4 self)
 Add to MetaCart
(Show Context)
Abstract. This paper presents a statistical model for textures that uses a nonnegative decomposition on a set of local atoms learned from an exemplar. This model is described by the variances and kurtosis of the marginals of the decomposition of patches in the learned dictionary. A fast sampling algorithm allows to draw a typical image from this model. The resulting texture synthesis captures the geometric features of the original exemplar. To speed up synthesis and generate structures of various sizes, a multiscale process is used. Applications to texture synthesis, image inpainting and texture segmentation are presented. 1 Statistical Models for Texture Synthesis The characterization of textures is a central topic in computer vision and graphics, mainly approached from a probabilistic point of view. Spatial domain modeling. The works of both Efros and Leung [1] and Wei and Levoy [2] pioneered a whole area of greedy approaches to texture synthesis. These methods copy pixels one by one, enforcing locally the consistence of the
Simultaneous codeword optimization (SimCO) for dictionary update and learning
 IEEE Trans. Signal Process
, 2012
"... Abstract—We consider the datadriven dictionary learning problem. The goal is to seek an overcomplete dictionary from which every training signal can be best approximated by a linear combination of only a few codewords. This task is often achieved by iteratively executing two operations: sparse cod ..."
Abstract

Cited by 13 (7 self)
 Add to MetaCart
Abstract—We consider the datadriven dictionary learning problem. The goal is to seek an overcomplete dictionary from which every training signal can be best approximated by a linear combination of only a few codewords. This task is often achieved by iteratively executing two operations: sparse coding and dictionary update. In the literature, there are two benchmark mechanisms to update a dictionary. The first approach, for example the MOD algorithm, is characterized by searching for the optimal codewords while fixing the sparse coefficients. In the second approach, represented by the KSVD method, one codeword and the related sparse coefficients are simultaneously updated while all other codewords and coefficients remain unchanged. We propose a novel framework that generalizes the aforementioned two methods. The unique feature of our approach is that one can update an arbitrary set of codewords and the corresponding sparse coefficients simultaneously: when sparse coefficients are fixed, the underlying optimization problem is the same as that in the MOD algorithm; when only one codeword is selected for update, it can be proved that the proposed algorithm is equivalent to the KSVD method; and more importantly, our method allows to update all codewords and all sparse coefficients simultaneously, hence the term simultaneously codeword optimization (SimCO). Under the proposed framework, we design two algorithms, namely the primitive and regularized SimCO. Simulations demonstrate that our approach excels the benchmark KSVD in terms of both learning performance and running speed. I.
Separable Dictionary Learning
"... Many techniques in computer vision, machine learning, and statistics rely on the fact that a signal of interest admits a sparse representation over some dictionary. Dictionaries are either available analytically, or can be learned from a suitable training set. While analytic dictionaries permit to c ..."
Abstract

Cited by 10 (6 self)
 Add to MetaCart
(Show Context)
Many techniques in computer vision, machine learning, and statistics rely on the fact that a signal of interest admits a sparse representation over some dictionary. Dictionaries are either available analytically, or can be learned from a suitable training set. While analytic dictionaries permit to capture the global structure of a signal and allow a fast implementation, learned dictionaries often perform better in applications as they are more adapted to the considered class of signals. In imagery, unfortunately, the numerical burden for (i) learning a dictionary and for (ii) employing the dictionary for reconstruction tasks only allows to deal with relatively small image patches that only capture local image information. The approach presented in this paper aims at overcoming these drawbacks by allowing a separable structure on the dictionary throughout the learning process. On the one hand, this permits larger patchsizes for the learning phase, on the other hand, the dictionary is applied efficiently in reconstruction tasks. The learning procedure is based on optimizing over a product of spheres which updates the dictionary as a whole, thus enforces basic dictionary properties such as mutual coherence explicitly during the learning procedure. In the special case where no separable structure is enforced, our method competes with stateoftheart dictionary learning methods like KSVD. 1.
A Nonconvex Relaxation Approach to Sparse Dictionary Learning
"... Dictionary learning is a challenging theme in computer vision. The basic goal is to learn a sparse representation from an overcomplete basis set. Most existing approaches employ a convex relaxation scheme to tackle this challenge due to the strong ability of convexity in computation and theoretical ..."
Abstract

Cited by 9 (3 self)
 Add to MetaCart
(Show Context)
Dictionary learning is a challenging theme in computer vision. The basic goal is to learn a sparse representation from an overcomplete basis set. Most existing approaches employ a convex relaxation scheme to tackle this challenge due to the strong ability of convexity in computation and theoretical analysis. In this paper we propose a nonconvex online approach for dictionary learning. To achieve the sparseness, our approach treats a socalled minimax concave (MC) penalty as a nonconvex relaxation of the ℓ0 penalty. This treatment expects to obtain a more robust and sparse representation than existing convex approaches. In addition, we employ an online algorithm to adaptively learn the dictionary, which makes the nonconvex formulation computationally feasible. Experimental results on the sparseness comparison and the applications in image denoising and image inpainting demonstrate that our approach is more effective and flexible. 1.
Learning sparse dictionaries for sparse signal representation
 IEEE Transactions on Signal Processing, (2008). submitted. CHAPTER 1. SPARSE COMPONENT ANALYSIS
"... An efficient and flexible dictionary structure is proposed for sparse and redundant signal representation. The structure is based on a sparsity model of the dictionary atoms over a base dictionary. The sparse dictionary provides efficient forward and adjoint operators, has a compact representation, ..."
Abstract

Cited by 7 (1 self)
 Add to MetaCart
(Show Context)
An efficient and flexible dictionary structure is proposed for sparse and redundant signal representation. The structure is based on a sparsity model of the dictionary atoms over a base dictionary. The sparse dictionary provides efficient forward and adjoint operators, has a compact representation, and can be effectively trained from given example data. In this, the sparse structure bridges the gap between implicit dictionaries, which have efficient implementations yet lack adaptability, and explicit dictionaries, which are fully adaptable but nonefficient and costly to deploy. In this report we discuss the advantages of sparse dictionaries, and present an efficient algorithm for training them. We demonstrate the advantages of the proposed structure for 3D image denoising. 1
Modeling Image Patches with a Generic Dictionary of MiniEpitomes
"... The goal of this paper is to question the necessity of features like SIFT in categorical visual recognition tasks. As an alternative, we develop a generative model for the raw intensity of image patches and show that it can support image classification performance on par with optimized SIFTbased ..."
Abstract

Cited by 7 (3 self)
 Add to MetaCart
(Show Context)
The goal of this paper is to question the necessity of features like SIFT in categorical visual recognition tasks. As an alternative, we develop a generative model for the raw intensity of image patches and show that it can support image classification performance on par with optimized SIFTbased techniques in a bagofvisualwords setting. Key ingredient of the proposed model is a compact dictionary of miniepitomes, learned in an unsupervised fashion on a large collection of images. The use of epitomes allows us to explicitly account for photometric and position variability in image appearance. We show that this flexibility considerably increases the capacity of the dictionary to accurately approximate the appearance of image patches and support recognition tasks. For image classification, we develop histogrambased image encoding methods tailored to the epitomic representation, as well as an “epitomic footprint ” encoding which is easy to visualize and highlights the generative nature of our model. We discuss in detail computational aspects and develop efficient algorithms to make the model scalable to large tasks. The proposed techniques are evaluated with experiments on the challenging PASCAL VOC 2007 image classification benchmark. 1.
Point Coding: Sparse Image Representation with Adaptive ShiftableKernel Dictionaries
"... Abstract—This paper addresses the problem of adaptively deriving optimally sparse image representations, using an dictionary composed of shiftable kernels. Algorithmic advantages of our solution make possible the computation of an approximately shiftinvariant adaptive image representation. Learned ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
(Show Context)
Abstract—This paper addresses the problem of adaptively deriving optimally sparse image representations, using an dictionary composed of shiftable kernels. Algorithmic advantages of our solution make possible the computation of an approximately shiftinvariant adaptive image representation. Learned kernels can have different sizes and adapt to different scales. Coefficient extraction uses a fast implementation of Matching Pursuit with essentially logarithmic cost per iteration. Dictionary update is performed by solving a structured leastsquares problem either by algebraic characterization of pseudoinverses of structured matrices, or by superfast interpolation methods. Kernels learned from natural images display expected 2D Gabor aspect (localization in orientation and frequency), as well as other structures commonly occurring in images (e.g., curved edges, or cross patterns), while when applied to newspaper text images, kernels tend to reproduce printed symbols or groups thereof. I.