Results 11  20
of
105
Principal surfaces from unsupervised kernel regression
 IEEE Transactions on Pattern Analysis and Machine Intelligence
, 2005
"... Abstract—We propose a nonparametric approach to learning of principal surfaces based on an unsupervised formulation of the NadarayaWatson kernel regression estimator. As compared with previous approaches to principal curves and surfaces, the new method offers several advantages: First, it provides ..."
Abstract

Cited by 26 (15 self)
 Add to MetaCart
(Show Context)
Abstract—We propose a nonparametric approach to learning of principal surfaces based on an unsupervised formulation of the NadarayaWatson kernel regression estimator. As compared with previous approaches to principal curves and surfaces, the new method offers several advantages: First, it provides a practical solution to the model selection problem because all parameters can be estimated by leaveoneout crossvalidation without additional computational cost. In addition, our approach allows for a convenient incorporation of nonlinear spectral methods for parameter initialization, beyond classical initializations based on linear PCA. Furthermore, it shows a simple way to fit principal surfaces in general feature spaces, beyond the usual data space setup. The experimental results illustrate these convenient features on simulated and real data. Index Terms—Dimensionality reduction, principal curves, principal surfaces, density estimation, model selection, kernel methods. æ 1
A KSegments Algorithm for Finding Principal Curves
 Pattern Recognition Letters
, 2000
"... We propose an incremental method to find principal curves. Line segments are fitted and connected to form polygonal lines. New segments are inserted until a performance criterion is met. Experimental results illustrate the performance of the method compared to other existing approaches. ..."
Abstract

Cited by 24 (2 self)
 Add to MetaCart
We propose an incremental method to find principal curves. Line segments are fitted and connected to form polygonal lines. New segments are inserted until a performance criterion is met. Experimental results illustrate the performance of the method compared to other existing approaches.
Locally defined principal curves and surfaces
 JOURNAL OF MACHINE LEARNING RESEARCH
, 2011
"... Principal curves are defined as selfconsistent smooth curves passing through the middle of the data, and they have been used in many applications of machine learning as a generalization, dimensionality reduction and a feature extraction tool. We redefine principal curves and surfaces in terms of th ..."
Abstract

Cited by 23 (0 self)
 Add to MetaCart
Principal curves are defined as selfconsistent smooth curves passing through the middle of the data, and they have been used in many applications of machine learning as a generalization, dimensionality reduction and a feature extraction tool. We redefine principal curves and surfaces in terms of the gradient and the Hessian of the probability density estimate. This provides a geometric understanding of the principal curves and surfaces, as well as a unifying view for clustering, principal curve fitting and manifold learning by regarding those as principal manifolds of different intrinsic dimensionalities. The theory does not impose any particular density estimation method can be used with any density estimator that gives continuous first and second derivatives. Therefore, we first present our principal curve/surface definition without assuming any particular density estimation method. Afterwards, we develop practical algorithms for the commonly used kernel density estimation (KDE) and Gaussian mixture models (GMM). Results of these algorithms are presented in notional data sets as well as real applications with comparisons to other approaches in the principal curve literature. All in all, we present a novel theoretical understanding of principal curves and surfaces, practical algorithms as general purpose machine learning tools, and applications of these algorithms to several practical problems.
Principal Curves: Learning, Design, And Applications
, 1999
"... The subjects of this thesis are unsupervised learning in general, and principal curves in particular. Principal curves were originally defined by Hastie \cite{Has84} and Hastie and Stuetzle \cite{HaSt89} (hereafter HS) to formally capture the notion of a smooth curve passing through the ``middle&apo ..."
Abstract

Cited by 18 (3 self)
 Add to MetaCart
The subjects of this thesis are unsupervised learning in general, and principal curves in particular. Principal curves were originally defined by Hastie \cite{Has84} and Hastie and Stuetzle \cite{HaSt89} (hereafter HS) to formally capture the notion of a smooth curve passing through the ``middle'' of a $d$dimensional probability distribution or data cloud. Based on the definition, HS also developed an algorithm for constructing principal curves of distributions and data sets. The field has been very active since Hastie and Stuetzle's groundbreaking work. Numerous alternative definitions and methods for estimating principal curves have been proposed, and principal curves were further analyzed and compared with other unsupervised learning techniques. Several applications in various areas including image analysis, feature extraction, and speech processing demonstrated that principal curves are not only of theoretical interest, but they also have a legitimate place in the family of practical unsupervised learning techniques. Although the concept of principal curves as considered by HS has several appealing characteristics, complete theoretical analysis of the model seems to be rather hard. This motivated us to redefine principal curves in a manner that allowed us to carry out extensive theoretical analysis while preserving the informal notion of principal curves. Our first contribution to the area is, hence, a new {\em theoretical model} that is analyzed by using tools of statistical learning theory. Our main result here is the first known consistency proof of a principal curve estimation scheme. The theoretical model proved to be too restrictive to be practical. However, it inspired the design of a new {\em practical algorithm} to estimate principal curves based on data. The polygonal line algorithm, which compares favorably with previous methods both in terms of performance and computational complexity, is our second contribution to the area of principal curves. To complete the picture, in the last part of the thesis we consider an {\em application} of the polygonal line algorithm to handwritten character skeletonization.
Elastic Principal Graphs and Manifolds and their Practical Applications
 COMPUTING
, 2005
"... Principal manifolds serve as useful tool for many practical applications. These manifolds are defined as lines or surfaces passing through “the middle” of data distribution. We propose an algorithm for fast construction of grid approximations of principal manifolds with given topology. It is based o ..."
Abstract

Cited by 17 (8 self)
 Add to MetaCart
Principal manifolds serve as useful tool for many practical applications. These manifolds are defined as lines or surfaces passing through “the middle” of data distribution. We propose an algorithm for fast construction of grid approximations of principal manifolds with given topology. It is based on analogy of principal manifold and elastic membrane. First advantage of this method is a form of the functional to be minimized which becomes quadratic at the step of the vertices position refinement. This makes the algorithm very effective, especially for parallel implementations. Another advantage is that the same algorithmic kernel is applied to construct principal manifolds of different dimensions and topologies. We demonstrate how flexibility of the approach allows numerous adaptive strategies like principal graph constructing, etc. The algorithm is implemented as a C++ package elmap and as a part of standalone data visualization tool VidaExpert, available on the web. We describe the approach and provide several examples of its application with speed performance characteristics.
Principal Curves With Bounded Turn
, 2002
"... Principal curves, like principal components, are a tool used in multivariate analysis for ends like feature extraction. Defined in their original form, principal curves need not exist for general distributions. The existence of principal curves with bounded length for any distribution that satisfies ..."
Abstract

Cited by 16 (0 self)
 Add to MetaCart
Principal curves, like principal components, are a tool used in multivariate analysis for ends like feature extraction. Defined in their original form, principal curves need not exist for general distributions. The existence of principal curves with bounded length for any distribution that satisfies some minimal regularity conditions has been shown. We define principal curves with bounded turn, show that they exist, and present a learning algorithm for them. Principal components are a special case of such curves when the turn is zero.
Sample complexity of testing the manifold hypothesis
 Advances in Neural Information Processing Systems 23
, 2010
"... The hypothesis that high dimensional data tends to lie in the vicinity of a low dimensional manifold is the basis of a collection of methodologies termed Manifold Learning. In this paper, we study statistical aspects of the question of fitting a manifold with a nearly optimal least squared error. Gi ..."
Abstract

Cited by 16 (1 self)
 Add to MetaCart
(Show Context)
The hypothesis that high dimensional data tends to lie in the vicinity of a low dimensional manifold is the basis of a collection of methodologies termed Manifold Learning. In this paper, we study statistical aspects of the question of fitting a manifold with a nearly optimal least squared error. Given upper bounds on the dimension, volume, and curvature, we show that Empirical Risk Minimization can produce a nearly optimal manifold using a number of random samples that is independent of the ambient dimension of the space in which data lie. We obtain an upper bound on the required number of samples that depends polynomially on the curvature, exponentially on the intrinsic dimension, and linearly on the intrinsic volume. For constant error, we prove a matching minimax lower bound on the sample complexity that shows that this dependence on intrinsic dimension, volume and curvature is unavoidable. Whether the known lower bound of O ( k ɛ2 1 log δ ɛ2) for the sample complexity of Empirical Risk minimization on k−means applied to data in a unit ball of arbitrary dimension is tight, has been an open question since 1997 [3]. Here ɛ is the desired bound on the error and δ is a bound on the probability of failure. We improve the best currently known upper bound [14] of k ɛ 2 min k,. Based on these results, we O ( k2 ɛ2 1 log δ ɛ2 k log4 ɛ) to O ɛ2 1 log δ ɛ2 devise a simple algorithm for k−means and another that uses a family of convex programs to fit a piecewise linear curve of a specified length to high dimensional data, where the sample complexity is independent of the ambient dimension. 1
Learning nonlinear image manifolds by global alignment of local linear models
, 2005
"... Appearance based methods, based on statistical models of the pixels values in an image (region) rather than geometrical object models, are increasingly popular in computer vision. In many applications the number of degrees of freedom (DOF) in the image generating process is much lower than the numb ..."
Abstract

Cited by 15 (0 self)
 Add to MetaCart
(Show Context)
Appearance based methods, based on statistical models of the pixels values in an image (region) rather than geometrical object models, are increasingly popular in computer vision. In many applications the number of degrees of freedom (DOF) in the image generating process is much lower than the number of pixels in the image. If there is a smooth function that maps the DOF to the pixel values, then the images are confined to a low dimensional manifold embedded in the image space. We propose a method based on probabilistic mixtures of factor analyzers to (i) model the density of images sampled from such manifolds and (ii) recover global parameterizations of the manifold. A globally nonlinear probabilistic twoway mapping between coordinates on the manifold and images is obtained by combining several, locally valid, linear mappings. We propose a parameter estimation scheme that improves upon an existing scheme, and experimentally compare the presented approach to selforganizing maps, generative topographic mapping, and mixtures of factor analyzers. In addition, we show that the approach also applies to find mappings between different embeddings of the same manifold.
Nonlinear dimensionality reduction with local spline embedding
 IEEE Trans. Knowl. Data Eng
, 2009
"... Abstract—This paper presents a new algorithm for Nonlinear Dimensionality Reduction (NLDR). Our algorithm is developed under the conceptual framework of compatible mapping. Each such mapping is a compound of a tangent space projection and a group of splines. Tangent space projection is estimated at ..."
Abstract

Cited by 12 (7 self)
 Add to MetaCart
(Show Context)
Abstract—This paper presents a new algorithm for Nonlinear Dimensionality Reduction (NLDR). Our algorithm is developed under the conceptual framework of compatible mapping. Each such mapping is a compound of a tangent space projection and a group of splines. Tangent space projection is estimated at each data point on the manifold, through which the data point itself and its neighbors are represented in tangent space with local coordinates. Splines are then constructed to guarantee that each of the local coordinates can be mapped to its own single global coordinate with respect to the underlying manifold. Thus, the compatibility between local alignments is ensured. In such a work setting, we develop an optimization framework based on reconstruction error analysis, which can yield a global optimum. The proposed algorithm is also extended to embed out of samples via spline interpolation. Experiments on toy data sets and realworld data sets illustrate the validity of our method. Index Terms—Nonlinear dimensionality reduction, compatible mapping, local spline embedding, out of samples. Ç 1