Results 1  10
of
12
HIGHDIMENSIONAL ESTIMATION WITH GEOMETRIC CONSTRAINTS
"... Abstract. Consider measuring a vector x ∈ Rn through the inner product with several measurement vectors, a1, a2,..., am. It is common in both signal processing and statistics to assume the linear response model yi = 〈ai, x〉+ εi, where εi is a noise term. However, in practice the precise relationshi ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
(Show Context)
Abstract. Consider measuring a vector x ∈ Rn through the inner product with several measurement vectors, a1, a2,..., am. It is common in both signal processing and statistics to assume the linear response model yi = 〈ai, x〉+ εi, where εi is a noise term. However, in practice the precise relationship between the signal x and the observations yi may not follow the linear model, and in some cases it may not even be known. To address this challenge, in this paper we propose a general model where it is only assumed that each observation yi may depend on ai only through 〈ai, x〉. We do not assume that the dependence is known. This is a form of the semiparametricsingle index model, and it includes the linear model as well as many forms of the generalized linear model as special cases. We further assume that the signal x has some structure, and we formulate this as a general assumption that x belongs to some known (but arbitrary) feasible set K ⊆ Rn. We carefully detail the benefit of using the signal structure to improve estimation. The theory is based on the mean width of K, a geometric parameter which can be used to understand its effective dimension in estimation problems. We determine a simple, efficient twostep procedure for estimating the signal based on this model – a linear estimation followed by metric projection onto K. We give general conditions under which the estimator is minimax optimal up to a constant. This leads to the intriguing conclusion that in the high noise regime, an unknown nonlinearity in the observations does not significantly reduce one’s ability to determine the signal, even when the nonlinearity may be noninvertible. Our results may be specialized to understand the effect of nonlinearities in compressed sensing. 1.
Subspace Embeddings and `pRegression Using Exponential Random Variables
, 2014
"... Oblivious lowdistortion subspace embeddings are a crucial building block for numerical linear algebra problems. We show for any real p, 1 ≤ p < ∞, given a matrix M ∈ Rn×d with n d, with constant probability we can choose a matrix Π with max(1, n1−2/p)poly(d) rows and n columns so that simultan ..."
Abstract

Cited by 3 (1 self)
 Add to MetaCart
Oblivious lowdistortion subspace embeddings are a crucial building block for numerical linear algebra problems. We show for any real p, 1 ≤ p < ∞, given a matrix M ∈ Rn×d with n d, with constant probability we can choose a matrix Π with max(1, n1−2/p)poly(d) rows and n columns so that simultaneously for all x ∈ Rd, ‖Mx‖p ≤ ‖ΠMx‖ ∞ ≤ poly(d)‖Mx‖p. Importantly, ΠM can be computed in the optimal O(nnz(M)) time, where nnz(M) is the number of nonzero entries of M. This generalizes all previous oblivious subspace embeddings which required p ∈ [1, 2] due to their use of pstable random variables. Using our matrices Π, we also improve the best known distortion of oblivious subspace embeddings of `1 into `1 with Õ(d) target dimension in O(nnz(M)) time from Õ(d3) to Õ(d2), which can further be improved to Õ(d3/2) log1/2 n if d = Ω(log n), answering a question of Meng and Mahoney (STOC, 2013). We apply our results to `pregression, obtaining a (1 + )approximation in O(nnz(M) log n) + poly(d/) time, improving the best known poly(d/) factors for every p ∈ [1,∞) \ {2}. If one is just interested in a poly(d) rather than a (1 + )approximation to `pregression, a corollary of our results is that for all p ∈ [1,∞) we can solve the `pregression problem without using general convex programming, that is, since our subspace embeds into ` ∞ it suffices to solve a linear programming problem. Finally, we give the first protocols for the distributed `pregression problem for every p ≥ 1 which are nearly optimal in communication and computation. 1
Subspace Embeddings and ℓpRegression Using Exponential Random Variables
, 2013
"... Oblivious lowdistortion subspace embeddings are a crucial building block for numerical linear algebra problems. We show for any real p, 1 ≤ p < ∞, given a matrix M ∈ R n×d with n ≫ d, with constant probability we can choose a matrix Π with max(1, n 1−2/p)poly(d) rows and n columns so that simult ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
Oblivious lowdistortion subspace embeddings are a crucial building block for numerical linear algebra problems. We show for any real p, 1 ≤ p < ∞, given a matrix M ∈ R n×d with n ≫ d, with constant probability we can choose a matrix Π with max(1, n 1−2/p)poly(d) rows and n columns so that simultaneously for all x ∈ R d, ‖Mx‖p ≤ ‖ΠMx‖ ∞ ≤ poly(d)‖Mx‖p. Importantly, ΠM can be computed in the optimal O(nnz(M)) time, where nnz(M) is the number of nonzero entries of M. This generalizes all previous oblivious subspace embeddings which required p ∈ [1, 2] due to their use of pstable random variables. Using our matrices Π, we also improve the best known distortion of oblivious subspace embeddings of ℓ1 into ℓ1 with Õ(d) target dimension in O(nnz(M)) time from Õ(d3) to Õ(d 2), which can further be improved to Õ(d3/2) log 1/2 n if d = Ω(log n), answering a question of Meng and Mahoney (STOC, 2013). We apply our results to ℓpregression, obtaining a (1 + ɛ)approximation in O(nnz(M) log n) + poly(d/ɛ) time, improving the best known poly(d/ɛ) factors for every p ∈ [1, ∞) \ {2}. If one is just interested in a poly(d) rather than a (1 + ɛ)approximation to ℓpregression, a corollary of our results is that for all p ∈ [1, ∞) we can solve the ℓpregression problem without using general convex programming, that is, since our subspace embeds into ℓ ∞ it suffices to solve a linear programming problem. Finally, we give the first protocols for the distributed ℓpregression problem for every p ≥ 1 which are nearly optimal in communication and computation. 1
Variable Selection in Nonparametric and Semiparametric Regression Models∗
, 2012
"... This chapter reviews the literature on variable selection in nonparametric and semiparametric regression models via shrinkage. We highlight recent developments on simultaneous variable selection and estimation through the methods of least absolute shrinkage and selection operator (Lasso), smoothly ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
This chapter reviews the literature on variable selection in nonparametric and semiparametric regression models via shrinkage. We highlight recent developments on simultaneous variable selection and estimation through the methods of least absolute shrinkage and selection operator (Lasso), smoothly clipped absolute deviation (SCAD) or their variants, but restrict our attention to nonparametric and semiparametric regression models. In particular, we consider variable selection in additive models, partially linear models, functional/varying coefficient models, single index models, general nonparametric regression models, and semiparametric/nonparametric quantile regression models.
MITCSAILTR2011041 CBCL303
, 2011
"... In this work we are interested in the problems of supervised learning and variable selection when the inputoutput dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key ide ..."
Abstract
 Add to MetaCart
In this work we are interested in the problems of supervised learning and variable selection when the inputoutput dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key idea is to measure the importance of each variable in the model by making use of partial derivatives. Based on this intuition we propose and study a new regularizer and a corresponding least squares regularization scheme. Using concepts and results from the theory of reproducing kernel Hilbert spaces and proximal methods, we show that the proposed learning algorithm corresponds to a minimization problem which can be provably solved by an iterative procedure. The consistency properties of the obtained estimator are studied both in terms of prediction and selection performance. An extensive empirical analysis shows that the proposed method performs favorably with respect to the stateoftheart. 1
Nonparametric Sparsity and Regularization
, 2011
"... In this work we are interested in the problems of supervised learning and variable selection when the inputoutput dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key ide ..."
Abstract
 Add to MetaCart
In this work we are interested in the problems of supervised learning and variable selection when the inputoutput dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key idea is to measure the importance of each variable in the model by making use of partial derivatives. Based on this intuition we propose and study a new regularizer and a corresponding least squares regularization scheme. Using concepts and results from the theory of reproducing kernel Hilbert spaces and proximal methods, we show that the proposed learning algorithm corresponds to a minimization problem which can be provably solved by an iterative procedure. The consistency properties of the obtained estimator are studied both in terms of prediction and selection performance. An extensive empirical analysis shows that the proposed method performs favorably with respect to the stateoftheart. 1
Statistical inference in compound functional models
"... Key words. Compound functional model, minimax estimation, sparse additive structure, dimension ..."
Abstract
 Add to MetaCart
(Show Context)
Key words. Compound functional model, minimax estimation, sparse additive structure, dimension
Submitted arXiv:math.ST/1106.4293v2 TIGHT CONDITIONS FOR CONSISTENCY OF VARIABLE SELECTION IN THE CONTEXT OF HIGH DIMENSIONALITY
, 2012
"... We address the issue of variable selection in the regression model with very high ambient dimension, i.e., when the number of variables is very large. The main focus is on the situation where the number of relevant variables, called intrinsic dimension and denoted by d ∗ , is much smaller than the a ..."
Abstract
 Add to MetaCart
(Show Context)
We address the issue of variable selection in the regression model with very high ambient dimension, i.e., when the number of variables is very large. The main focus is on the situation where the number of relevant variables, called intrinsic dimension and denoted by d ∗ , is much smaller than the ambient dimension d. Without assuming any parametric form of the underlying regression function, we get tight conditions making it possible to consistently estimate the set of relevant variables. These conditions relate the intrinsic dimension to the ambient dimension and to the sample size. The procedure that is provably consistent under these tight conditions is based on comparing quadratic functionals of the empirical Fourier coefficients with appropriately chosen threshold values. The asymptotic analysis reveals the presence of two quite different regimes. The first regime is when d ∗ is fixed. In this case the situation in nonparametric regression is the same as in linear regression, i.e., consistent variable selection is possible if and only if logd is small compared to the sample size n. The picture is different in the second regime, d ∗ →∞as n→∞, where we prove that consistent variable selection in nonparametric setup is possible only if d ∗ + loglogd is small compared to logn. We apply these results to derive minimax separation rates for the problem of variable selection. 1. Introduction. Realworld
Electronic Journal of Statistics
"... Minimax testing of a composite null hypothesis defined via a quadratic functional in the model of regression ..."
Abstract
 Add to MetaCart
(Show Context)
Minimax testing of a composite null hypothesis defined via a quadratic functional in the model of regression