Results 1  10
of
12
MATRIX CONCENTRATION INEQUALITIES VIA THE METHOD OF EXCHANGEABLE PAIRS
 SUBMITTED TO THE ANNALS OF PROBABILITY
, 2013
"... This paper derives exponential concentration inequalities and polynomial moment inequalities for the spectral norm of a random matrix. The analysis requires a matrix extension of the scalar concentration theory developed by Sourav Chatterjee using Stein’s method of exchangeable pairs. Whenapplied to ..."
Abstract

Cited by 18 (4 self)
 Add to MetaCart
This paper derives exponential concentration inequalities and polynomial moment inequalities for the spectral norm of a random matrix. The analysis requires a matrix extension of the scalar concentration theory developed by Sourav Chatterjee using Stein’s method of exchangeable pairs. Whenapplied toasum of independentrandom matrices, this approach yields matrix generalizations of the classical inequalities due to Hoeffding, Bernstein, Khintchine, and Rosenthal. The same technique delivers bounds for sums of dependent random matrices and more general matrixvalued functions of dependent random variables. This paper is based on two independent manuscripts from mid2011 that both applied the method of exchangeable pairs to establish matrix concentration inequalities. One manuscript is by Mackey and Jordan; the other is by Chen, Farrell, and Tropp. The authors have combined this research into a single unified presentation, with equal contributions from both groups.
Improved matrix algorithms via the subsampled randomized Hadamard transform
 SIAM J. Matrix Analysis Applications
"... Abstract. Several recent randomized linear algebra algorithms rely upon fast dimension reduction methods. A popular choice is the subsampled randomized Hadamard transform (SRHT). In this article, we address the efficacy, in the Frobenius and spectral norms, of an SRHTbased lowrank matrix approxim ..."
Abstract

Cited by 17 (3 self)
 Add to MetaCart
(Show Context)
Abstract. Several recent randomized linear algebra algorithms rely upon fast dimension reduction methods. A popular choice is the subsampled randomized Hadamard transform (SRHT). In this article, we address the efficacy, in the Frobenius and spectral norms, of an SRHTbased lowrank matrix approximation technique introduced by Woolfe, Liberty, Rohklin, and Tygert. We establish a slightly better Frobenius norm error bound than is currently available, and a much sharper spectral norm error bound (in the presence of reasonable decay of the singular values). Along the way, we produce several results on matrix operations with SRHTs (such as approximate matrix multiplication) that may be of independent interest. Our approach builds upon Tropp’s in “Improved Analysis of the
Linear convergence with condition number independent access of full gradients
 Advances in Neural Information Processing Systems
, 2013
"... For smooth and strongly convex optimizations, the optimal iteration complexity of the gradientbased algorithm is O( κ log 1/ǫ), where κ is the condition number. In the case that the optimization problem is illconditioned, we need to evaluate a large number of full gradients, which could be computa ..."
Abstract

Cited by 13 (1 self)
 Add to MetaCart
(Show Context)
For smooth and strongly convex optimizations, the optimal iteration complexity of the gradientbased algorithm is O( κ log 1/ǫ), where κ is the condition number. In the case that the optimization problem is illconditioned, we need to evaluate a large number of full gradients, which could be computationally expensive. In this paper, we propose to remove the dependence on the condition number by allowing the algorithm to access stochastic gradients of the objective function. To this end, we present a novel algorithm named Epoch Mixed Gradient Descent (EMGD) that is able to utilize two kinds of gradients. A distinctive step in EMGD is the mixed gradient descent, where we use a combination of the full and stochastic gradients to update the intermediate solution. Theoretical analysis shows that EMGD is able to find an ǫoptimal solution by computing O(log 1/ǫ) full gradients and O(κ2 log 1/ǫ) stochastic gradients. 1
The effect of coherence on sampling from matrices with orthonormal columns, and preconditioned least squares problems. arXiv preprint arXiv:1203.4809
, 2012
"... ar ..."
(Show Context)
Recovering the Optimal Solution by Dual Random Projection
"... Random projection has been widely used in data classification. It maps highdimensional data into a lowdimensional subspace in order to reduce the computational cost in solving the related optimization problem. While previous studies are focused on analyzing the classification performance of using ..."
Abstract

Cited by 7 (5 self)
 Add to MetaCart
Random projection has been widely used in data classification. It maps highdimensional data into a lowdimensional subspace in order to reduce the computational cost in solving the related optimization problem. While previous studies are focused on analyzing the classification performance of using random projection, in this work, we consider the recovery problem, i.e., how to accurately recover the optimal solution to the original optimization problem in the highdimensional space based on the solution learned from the subspace spanned by random projections. We present a simple algorithm, termed Dual Random Projection, that uses the dual solution of the lowdimensional optimization problem to recover the optimal solution to the original problem. Our theoretical analysis shows that with a high probability, the proposed algorithm is able to accurately recover the optimal solution to the original problem, provided that the data matrix is of low rank or can be well approximated by a low rank matrix.
TANGENT SPACE ESTIMATION FOR SMOOTH EMBEDDINGS OF RIEMANNIAN MANIFOLDS
"... Abstract. Numerous dimensionality reduction problems in data analysis involve the recovery of lowdimensional models or the learning of manifolds underlying sets of data. Many manifold learning methods require the estimation of the tangent space of the manifold at a point from locally available data ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
Abstract. Numerous dimensionality reduction problems in data analysis involve the recovery of lowdimensional models or the learning of manifolds underlying sets of data. Many manifold learning methods require the estimation of the tangent space of the manifold at a point from locally available data samples. Local sampling conditions such as (i) the size of the neighborhood (sampling width) and (ii) the number of samples in the neighborhood (sampling density) affect the performance of learning algorithms. In this work, we propose a theoretical analysis of local sampling conditions for the estimation of the tangent space at a point P lying on a mdimensional Riemannian manifold S in Rn. Assuming a smooth embedding of S in Rn, we estimate the tangent space TP S by performing a Principal Component Analysis (PCA) on points sampled from the neighborhood of P on S. Our analysis explicitly takes into account the second order properties of the manifold at P, namely the principal curvatures as well as the higher order terms. We consider a random sampling framework and leverage recent results from random matrix theory to derive conditions on the sampling width and the local sampling density for an accurate estimation of tangent subspaces. We measure the estimation accuracy by the angle between the estimated tangent space TP S and the true tangent space TP S and we give conditions for this angle to be bounded with high probability. In particular, we observe that the local sampling conditions are highly dependent on the correlation between the components in the secondorder local approximation of the manifold. We finally provide numerical simulations to validate our theoretical findings. 1.
Information and Inference: A Journal of the IMA (2013) 2, 69–114 doi:10.1093/imaiai/iat003 Tangent space estimation for smooth embeddings of Riemannian manifolds R
"... Numerous dimensionality reduction problems in data analysis involve the recovery of lowdimensional models or the learning of manifolds underlying sets of data. Many manifold learning methods require the estimation of the tangent space of the manifold at a point from locally available data samples. ..."
Abstract
 Add to MetaCart
(Show Context)
Numerous dimensionality reduction problems in data analysis involve the recovery of lowdimensional models or the learning of manifolds underlying sets of data. Many manifold learning methods require the estimation of the tangent space of the manifold at a point from locally available data samples. Local sampling conditions such as (i) the size of the neighborhood (sampling width) and (ii) the number of samples in the neighborhood (sampling density) affect the performance of learning algorithms. In this work, we propose a theoretical analysis of local sampling conditions for the estimation of the tangent space at a point P lying on an mdimensional Riemannian manifold S in Rn. Assuming a smooth embedding of S in Rn, we estimate the tangent space TPS by performing a principal component analysis (PCA) on points sampled from the neighborhood of P on S. Our analysis explicitly takes into account the secondorder properties of the manifold at P, namely the principal curvatures as well as the higherorder terms. We consider a random sampling framework and leverage recent results from random matrix theory to derive conditions on the sampling width and the local sampling density for an accurate estimation of tangent subspaces. We measure the estimation accuracy by the angle between the estimated tangent space T̂PS and the true tangent space TPS and we give conditions for this angle to be bounded with high probability. In particular, we observe that the local sampling conditions are highly dependent on the correlation between the components in the secondorder local approximation of the manifold. We finally provide numerical simulations to validate our theoretical findings.
Random Projections for Classification: A Recovery Approach
"... Abstract — Random projection has been widely used in data classification. It maps highdimensional data into a lowdimensional subspace in order to reduce the computational cost in solving the related optimization problem. While previous studies are focused on analyzing the classification performanc ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract — Random projection has been widely used in data classification. It maps highdimensional data into a lowdimensional subspace in order to reduce the computational cost in solving the related optimization problem. While previous studies are focused on analyzing the classification performance in the lowdimensional space, in this paper, we consider the recovery problem, i.e., how to accurately recover the optimal solution to the original highdimensional optimization problem based on the solution learned after random projection. We present a simple algorithm, termed dual random projection, which uses the dual solution of the lowdimensional optimization problem to recover the optimal solution to the original problem. Our theoretical analysis shows that with a high probability, the proposed algorithm is able to accurately recover the optimal solution to the original problem, provided that the data matrix is (approximately) lowrank and/or optimal solution is (approximately) sparse. We further show that the proposed algorithm can be applied iteratively to reducing the recovery error exponentially. Index Terms — Random projection, primal solution, dual solution, lowrank, sparse.