Results 1  10
of
14
Simultaneous codeword optimization (SimCO) for dictionary update and learning
 IEEE Trans. Signal Process
, 2012
"... Abstract—We consider the datadriven dictionary learning problem. The goal is to seek an overcomplete dictionary from which every training signal can be best approximated by a linear combination of only a few codewords. This task is often achieved by iteratively executing two operations: sparse cod ..."
Abstract

Cited by 14 (8 self)
 Add to MetaCart
(Show Context)
Abstract—We consider the datadriven dictionary learning problem. The goal is to seek an overcomplete dictionary from which every training signal can be best approximated by a linear combination of only a few codewords. This task is often achieved by iteratively executing two operations: sparse coding and dictionary update. In the literature, there are two benchmark mechanisms to update a dictionary. The first approach, for example the MOD algorithm, is characterized by searching for the optimal codewords while fixing the sparse coefficients. In the second approach, represented by the KSVD method, one codeword and the related sparse coefficients are simultaneously updated while all other codewords and coefficients remain unchanged. We propose a novel framework that generalizes the aforementioned two methods. The unique feature of our approach is that one can update an arbitrary set of codewords and the corresponding sparse coefficients simultaneously: when sparse coefficients are fixed, the underlying optimization problem is the same as that in the MOD algorithm; when only one codeword is selected for update, it can be proved that the proposed algorithm is equivalent to the KSVD method; and more importantly, our method allows to update all codewords and all sparse coefficients simultaneously, hence the term simultaneously codeword optimization (SimCO). Under the proposed framework, we design two algorithms, namely the primitive and regularized SimCO. Simulations demonstrate that our approach excels the benchmark KSVD in terms of both learning performance and running speed. I.
From MAP to marginals: Variational inference in Bayesian submodular models
 In Neural Information Processing Systems (NIPS
, 2014
"... Submodular optimization has found many applications in machine learning and beyond. We carry out the first systematic investigation of inference in probabilistic models defined through submodular functions, generalizing regular pairwise MRFs and Determinantal Point Processes. In particular, we pres ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
(Show Context)
Submodular optimization has found many applications in machine learning and beyond. We carry out the first systematic investigation of inference in probabilistic models defined through submodular functions, generalizing regular pairwise MRFs and Determinantal Point Processes. In particular, we present LFIELD, a variational approach to general logsubmodular and logsupermodular distributions based on sub and supergradients. We obtain both lower and upper bounds on the logpartition function, which enables us to compute probability intervals for marginals, conditionals and marginal likelihoods. We also obtain fully factorized approximate posteriors, at the same computational cost as ordinary submodular optimization. Our framework results in convex problems for optimizing over differentials of submodular functions, which we show how to optimally solve. We provide theoretical guarantees of the approximation quality with respect to the curvature of the function. We further establish natural relations between our variational approach and the classical meanfield method. Lastly, we empirically demonstrate the accuracy of our inference scheme on several submodular models. 1
Analyzing the subspace structure of related images: Concurrent segmentation of image sets
 In ECCV
, 2012
"... Abstract. We develop new algorithms to analyze and exploit the joint subspace structure of a set of related images to facilitate the process of concurrent segmentation of a large set of images. Most existing approaches for this problem are either limited to extracting a single similar object across ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
(Show Context)
Abstract. We develop new algorithms to analyze and exploit the joint subspace structure of a set of related images to facilitate the process of concurrent segmentation of a large set of images. Most existing approaches for this problem are either limited to extracting a single similar object across the given image set or do not scale well to a large number of images containing multiple objects varying at different scales. One of the goals of this paper is to show that various desirable properties of such an algorithm (ability to handle multiple images with multiple objects showing arbitary scale variations) can be cast elegantly using simple constructs from linear algebra: this significantly extends the operating range of such methods. While intuitive, this formulation leads to a hard optimization problem where one must perform the image segmentation task together with appropriate constraints which enforce desired algebraic regularity (e.g., common subspace structure). We propose efficient iterative algorithms (with small computational requirements) whose key steps reduce to objective functions solvable by maxflow and/or nearly closed form identities. We study the qualitative, theoretical, and empirical properties of the method, and present results on benchmark datasets. 1
Scalable variational inference in logsupermodular models
, 2015
"... We consider the problem of approximate Bayesian inference in logsupermodular models. These models encompass regular pairwise MRFs with binary variables, but allow to capture highorder interactions, which are intractable for existing approximate inference techniques such as belief propagation, mean ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
(Show Context)
We consider the problem of approximate Bayesian inference in logsupermodular models. These models encompass regular pairwise MRFs with binary variables, but allow to capture highorder interactions, which are intractable for existing approximate inference techniques such as belief propagation, mean field, and variants. We show that a recently proposed variational approach to inference in logsupermodular models –LFIELD – reduces to the widelystudied minimum norm problem for submodular minimization. This insight allows to leverage powerful existing tools, and hence to solve the variational problem orders of magnitude more efficiently than previously possible. We then provide another natural interpretation of LFIELD, demonstrating that it exactly minimizes a specific type of Rényi divergence measure. This insight sheds light on the nature of the variational approximations produced by LFIELD. Furthermore, we show how to perform parallel inference as message passing in a suitable factor graph at a linear convergence rate, without having to sum up over all the configurations of the factor. Finally, we apply our approach to a challenging image segmentation task. Our experiments confirm scalability of our approach, high quality of the marginals, and the benefit of incorporating higherorder potentials.
Causal meets Submodular: Subset Selection with Directed Information
"... Abstract We study causal subset selection with Directed Information as the measure of prediction causality. Two typical tasks, causal sensor placement and covariate selection, are correspondingly formulated into cardinality constrained directed information maximizations. To attack the NPhard probl ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract We study causal subset selection with Directed Information as the measure of prediction causality. Two typical tasks, causal sensor placement and covariate selection, are correspondingly formulated into cardinality constrained directed information maximizations. To attack the NPhard problems, we show that the first problem is submodular while not necessarily monotonic. And the second one is "nearly" submodular. To substantiate the idea of approximate submodularity, we introduce a novel quantity, namely submodularity index (SmI), for general set functions. Moreover, we show that based on SmI, greedy algorithm has performance guarantee for the maximization of possibly nonmonotonic and nonsubmodular functions, justifying its usage for a much broader class of problems. We evaluate the theoretical results with several case studies, and also illustrate the application of the subset selection to causal structure learning.
A Fast Greedy Algorithm for Generalized Column Subset Selection
"... This paper defines a generalized column subset selection problem which is concerned with the selection of a few columns from a source matrix A that best approximate the span of a target matrix B. The paper then proposes a fast greedy algorithm for solving this problem and draws connections to diff ..."
Abstract
 Add to MetaCart
(Show Context)
This paper defines a generalized column subset selection problem which is concerned with the selection of a few columns from a source matrix A that best approximate the span of a target matrix B. The paper then proposes a fast greedy algorithm for solving this problem and draws connections to different problems that can be efficiently solved using the proposed algorithm. 1 Generalized Column Subset Selection The Column Subset Selection (CSS) problem can be generally defined as the selection of a few columns from a data matrix that best approximate its span [2–5,10,15]. We extend this definition to the generalized problem of selecting a few columns from a source matrix to approximate the span of a target matrix. The generalized CSS problem can be formally defined as follows: Problem 1 (Generalized Column Subset Selection) Given a source matrix A ∈ Rm×n, a target matrix B ∈ Rm×r and an integer l, find a subset of columns L from A such that L  = l and L = arg minS ‖B − P (S)B‖2F, where S is the set of the indices of the candidate columns from A, P (S) ∈ Rm×m is a projection matrix which projects the columns of B onto the span of the set S of columns, and L is the set of the indices of the selected columns from A. The CSS criterion F (S) = ‖B −P (S)B‖2F represents the sum of squared errors between the target matrix B and its rankl approximation P (S)B. In other words, it calculates the Frobenius norm of the residual matrix F = B − P (S)B. Other types of matrix norms can also be used to quantify the reconstruction error [2, 3]. The present work, however, focuses on developing algorithms that minimize the Frobenius norm of the residual matrix. The projection matrix P (S) can be calculated as P (S) = A:S AT:SA:S
Adaptive Submodular Dictionary Selection for Sparse Representation Modeling with Application to Image SuperResolution
"... This paper proposes an adaptive dictionary learning approach based on submodular optimization. With the lowfrequency components by the analytic DCT atoms, highresolution dictionaries can be inferred through online learning to make efficient approximation with rapid convergence. It is formulated a ..."
Abstract
 Add to MetaCart
(Show Context)
This paper proposes an adaptive dictionary learning approach based on submodular optimization. With the lowfrequency components by the analytic DCT atoms, highresolution dictionaries can be inferred through online learning to make efficient approximation with rapid convergence. It is formulated as a combinatorial optimization for approximate submodularity, which is suitable for sparse representation based on dictionaries with arbitrary structures. With application to singleimage superresolution, the proposed scheme has been demonstrated to outperform the double sparsity dictionary in reconstruction quality and operate faster than standard KSVD. Given a collection of training examples Y = {y1, · · · , yN}, a dictionary D is constructed by selecting atoms from the combination of the trained and analytic dictionaries. DCT dictionary with a fast computation is adopted as the analytic component, while online dictionary learning based on stochastic approximation is employed to improve computational efficiency. Both the selection of dictionary columns and the sparse representation of signals are formulated as a combinatorial optimization problem Ls(A) = min w
SUBMODULAR VOLUME SIMPLEX ANALYSIS: A GREEDY ALGORITHM FOR HYPERSPECTRAL UNMIXNG
"... The hyperspectral unmixing problem can be formulated as a combinatorial optimization which selects the spectral vectors that maximize the volume of a simplex, with the assumptions that the dataset contain pure pixels and the mixture is linear. Submodularity presents an intuitive diminishing returns ..."
Abstract
 Add to MetaCart
(Show Context)
The hyperspectral unmixing problem can be formulated as a combinatorial optimization which selects the spectral vectors that maximize the volume of a simplex, with the assumptions that the dataset contain pure pixels and the mixture is linear. Submodularity presents an intuitive diminishing returns property which arises naturally in discrete and combinatorial optimization problems. Submodular functions enable the application of fast, greedy algorithms which possess near optimal approximation guarantees. This paper proposes a submodular greedybased approach for solving the spectral unmixing problem by modifying the objective function to become a nondecreasing submodular function. Theoretical and experimental results are presented to demonstrate the feasibility of the method. Index Terms — Submodular optimization, greedy algorithm, hyperspectral unmixing, endmember extraction