S. Roweis, and Z. Ghahramani. "A Unifying Review of Linear Gaussian Models, " Neural Computation, 11(2):305--345, 1999.  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... an in the framework of mixed BNs (BNs that have a mixture of continuous and discrete variables) The BN formalism has previously been presented as a statistical pattern recognition framework that is more generic than that of HMMs [10] That is, while they are in the same family of models [9], BNs are more general in that they provide more exibility in changing the topology of the model and, hence, the structure of the component distributions. With this exibility, we address two questions: 1. Should the distribution for an itself be conditioned upon q n : p(a n jq n ) or be left ....

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2), 1999.

....assignments to A: P (XjM) K k=1 P (X; Ak jM) 7) 3 Model Setup 3.1 Dynamic Bayesian Networks 3.1.1 De nition We have set up our models within the framework of dynamic Bayesian networks (DBN s) 4, 5] as opposed to the conventional HMM s. DBN s are in the same family of models as HMM s [6], but using them speci cally allows more ease in experimenting with di erent topologies and with hidden vs. observable data. 7, 8] provide the foundation for how we do ASR with DBN s. A BN, of which a DBN is a speci c type, has the following components: 1. A set of variables Z to model 2. A ....

S. Roweis and Z. Ghahramani, \A unifying review of linear Gaussian models," Neural Computation, vol. 11, no. 2, 1999.

....a vector. Points that do not adjust well to the clustering model are called outliers. K means uses Euclidean distance; the distance from t i to C j is (t i ; C j ) t i C j ) t i C j ) 1) 2. 2 The K means clustering algorithm K means [19] is one of the most popular clustering algorithms [6, 10, 24]. It is simple and fairly fast [9, 6] K means is initialized from some random or approximate solution. Each iteration assigns each point to its nearest cluster and then points belonging to the same cluster are averaged to get new cluster centroids. Each iteration successively improves cluster ....

....where C j is the nearest cluster centroid of t i . The quality of a clustering model is measured by the the sum of squared distances from each point to the cluster where it was assigned [26, 21, 6] This quantity is proportional to the average quantization error, also known as distortion [19, 24]. The quality of a solution is measured as: q(C) t i ; C j ) 3) which can be computed from R as q(R; W ) j=1 W j l=1 R lj : 4) 2.3 Example We now present an example with d = 16; k = 4 to motivate the need for a summary table. Assume items come as transactions containing ....

S. Roweis and Z. Ghahramani. A unifying review of Linear Gaussian Models. Neural Computation, 1999.

....maximize the likelihood of the observed data vectors. The solution can be obtained by eigenvector decomposition of a sample covariance matrix. In factor analysis similar to PCA, a diagonal covariance is assumed for the noise model. Comparison between factor analysis and PCA can be found in [5][6]. However, PPCA does not give the optimal number of independent components because it uses a maximum likelihood criterion, thus leading to the over fitting problem. Bayesian PCA [7] 8] was proposed to find the intrinsic dimension and the optimal number of clusters in the latent variable model by ....

S. Roweis, Z. Ghahramani, "A unifying review of linear Gaussian models," Neural computation 11, 1999.

....models and deformable models. These are to a large extent complementary and can use well established statistical learning theory in either o ine or online learning. A unifying review of theory and techniques for such models from the machine learning perspective is given in Roweis and Ghahramani [5]. They explain the relationship between the models and even o er a generative model for generative models. Table I gives an overview of how the models can be regarded as extensions of each other. Markov models and Bayesian networks are generally regarded as graphical models (for example Figure ....

S. Roweis and Z. Ghahramani, \A unifying review of linear Gaussian models," Neural Computation, vol. 11, pp. 305-345, 1999.

....dimension (factor) This paper introduces an extension to the standard factor analysis which is applicable to HMMs. The model is called factor analysed HMM (FAHMM) FAHMMs belong to a broad class of generalised linear Gaussian models [16] which extends the set of standard linear Gaussian models [17]. Generalised linear Gaussian models are state space models with linear state evolution and observation processes, and Gaussian mixture distributed noise processes. The underlying HMM generates piecewise constant state vector trajectories that are mapped into the observation space via linear ....

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1999.

....usually cannot be inferred from human knowledge about the data domain and therefore the model building process is usually driven by computational considerations. Although our approach can be interpreted in terms of a generative model of the data, in contrast to most other generative models (see [10] for an overview) the present approach is nonparametric, since no specific assumptions about the functional form of the data distribution have to be made. In that way our approach compares well with other quantization methods, like principal curves and surfaces [4, 13, 6] which only have to make ....

....b( as in (11) Thereby the components of w i contain the quantization levels of the i th coordinate axis with direction a i . Further, it makes sense to normalize the direction vectors according to ka i k = 1. There are strong similarities with a parametric ICA model which has been suggested in [10], where source densities have been mixtures of delta functions and additive noise has been isotropic Gaussian. Other unsupervised learning methods which correspond to different projection sets, like principal curves or multilayer perceptrons (see [7] for an overview) can as well be incorporated ....

Sam Roweis and Zoubin Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1999.

....compared against the standard EM algorithm and the On Line EM algorithm. 1 Introduction Clustering is one of the most important data mining [12] techniques used nowadays. This problem has been extensively studied by the statistics [9, 29, 32] database [1, 5, 11, 17, 23, 36] and machine learning [10, 18, 30, 34] communities. Clustering algorithms partition a data set into several groups such that points in the same group are close to each other and points across groups are far from each other [10] Most algorithms work with numeric data [3, 5, 14, 34, 36] but there is some recent work on clustering ....

....dimensional data has been the approach in [1, 2, 3] Sampling and choosing representative points is proposed in [14] 1. 1 The EM Algorithm We present a fast and robust clustering algorithm for high dimensional and large data sets based on the well known Expectation Maximization (EM) algorithm [6, 8, 25, 30, 34, 35]. The EM algorithm is a general statistical method of maximum likelihood estimation [8, 34] and in particular it can be used to perform clustering. # This work was partially conducted at the Georgia Institute of Technology during PhD studies. The EM algorithm has many desirable features [34, ....

[Article contains additional citation context not shown here]

S. Roweis and Z. Ghahramani. A unifying review of Linear Gaussian Models. Neural Computation, 1999.

....setting by Baker ( 8] and Jelinek et al. ( 61, 6, 63, 62, 10, 64, 7] See Rabiner( 97] for an introduction. Hidden Markov models have connections to lots of similar models which treat linear systems with Gaussian noise: Kalman filters, PCA etc. For a unified treatment, see Roweis and Ghahramani([102]) The connection between HMM s and Kalman filters was earlier elucidated by Digalakis et al. ( 36] Let us first consider a system which is described at any (discrete) time as being in one of a set of N distinct states: S1, S2, SN) and which undergoes state transitions at regularly spaces ....

S. Roweis and Z. Ghahramani, "A Unifying Review of Linear Gaussian Mod- els", Neural Computation, vol. 11, pp. 305 345, 1999.

....38, 6] The main limitation of the BMA approach is that variables are required to be discrete. Fortunately, in many applications the data can be discretized. Also, it is possible to extend our work to continuous (e.g. Gaussian) variables (for a review of continuous variable Bayesian network, see [36]) 7. Conclusion In this paper we have described a framework for morphology function analysis, based on a Bayesian network model of relationships among image and functional variables. The algorithms based on this framework generate sets of voxels whose members have similar probabilistic ....

Roweis, S., and Ghahramani, Z., "A unifying review of linear Gaussian models," Neural Computation, vol.11, no.2, pp.305-345, 1999.

....to overcome this problem [29, 40] In addition to correlation modelling they address the problem of high dimensionality of the feature vectors by allowing lower dimensional subspaces to be used. The machine learning community has been interested in linear Gaussian models (LGM) for some time now [38] due to the eciency and applicability of the expectation maximisation (EM) algorithm [5] which provides a consistent framework in supervised learning. Despite the attempts to unify the eld of LGMs, several interesting models have been omitted; e.g. independent factor analysis [1] linear ....

....Gaussian models are a subset of more general state space models that consist of state evolution process and observation process. Strictly speaking linear Gaussian models are state space models in which the state evolution and observation equations are linear and the distributions are Gaussians [38]. Some models presented in this paper deviate from the strict de nition in that mixtures of Gaussians are allowed as the distributions. Therefore they are called generalised LGMs. This generalisation is important in case of feature vectors which tend to have multi modal distributions due to ....

[Article contains additional citation context not shown here]

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305-345, 1999.

.... a closed form solution) is only possible in a very limited set of cases, most notably when all hidden nodes are discrete, or when all nodes (hidden and observed) have linear Gaussian distributions (in which case the network is just a sparse parameterization of a joint multivariate Gaussian [SK89, RG99] Expert systems and hidden Markov models (HMMs) fall in the former category, while factor analysis and Kalman lters fall in the latter. There are two main kinds of exact inference algorithms: those that only work on DAG models, and those that work on directed and undirected graphs. 4 The ....

S. Roweis and Z. Ghahramani. A Unifying Review of Linear Gaussian Models. Neural Computation, 11(2), 1999.

....the equation: x = As (1) where the matrix A 2 R m n is called the mixing matrix and 2 R m is iid (independent, identically distributed) gaussian noise. We want to estimate the sources s on the basis of the observations x. This framework is closely related to the classical Kalman lter [35, 62]. The coding formulation is as follows: we want to encode the image x into a linear expansion c 2 R n such that x = Bc r (2) A shorter version of this report has been submitted for publication in the special issue on Statistics of Shapes and Textures of the Journal of Mathematical Imaging ....

....and pointwise (see subsection 2.5) It is interesting to contrast Table 1 with a similar table of learning methods: Table 2: Learning Methods constrained low noise constrained noisy under constrained sub gaussian ICA a IFA b (none) c gaussian PCA, ZCA, etc. d factor analysis (e.g. [62]) none) c super gaussian ICA a IFA b ICA extensions [31] a ICA: independent component analysis [9, 32] b IFA: independent factor analysis [1] c No methods are possible for this case (see section 4) d PCA: principal component analysis (see e.g. 62] ZCA: zero phase component ....

[Article contains additional citation context not shown here]

S. Roweis, Z. Ghahramani, \A Unifying Review of Linear Gaussian Models", Neural Computation vol.11, no.2, pp.305-345, 1999.

....to overcome this problem [29, 40] In addition to correlation modelling they address the problem of high dimensionality of the feature vectors by allowing lower dimensional subspaces to be used. The machine learning community has been interested in linear Gaussian models (LGM) for some time now [38] due to the eciency and applicability of the expectation maximisation (EM) algorithm [5] which provides a consistent framework in supervised learning. Despite the attempts to unify the eld of LGMs, several interesting models have been omitted; e.g. independent factor analysis [1] linear ....

....Gaussian models are a subset of more general state space models that consist of state evolution process and observation process. Strictly speaking linear Gaussian models are state space models in which the state evolution and observation equations are linear and the distributions are Gaussians [38]. Some models presented in this paper deviate from the strict de nition in that mixtures of Gaussians are allowed as the distributions. Therefore they are called generalised LGMs. This generalisation is important in case of feature vectors which tend to have multimodal distributions due to source ....

[Article contains additional citation context not shown here]

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305-345, 1999.

....(or semantics) from raw signals, and thus can learns hierarchical models. Examples of generative method are principle component analysis (PCA) independent component analysis (ICA) transformed component analysis (TCA) 2] image coding[9] and hidden Markov models (HMM) As a recent review paper[10] pointed out, existing generative models mentioned above su er from the simpli ed assumption that hidden variables are independent and identically distributed. Therefore they are not powerful enough to model realistic visual patterns. For example, an image coding model cannot synthesize a texture ....

S. Roweis and Z. Ghahramani, \A unifying review of linear Gaussian models", Neural Computation, vol. 11, no. 2, 1999.

....divisible (see section A of the appendix) 8 Note that this is not the same as the Laplace approximation of the integral. The latter would require nding the mode of the log integrand which is di erent from . The approximation applied here is also used in Kalman lters or smoothers (e.g. [18]) 4 MUTUAL INFORMATION KERNELS FOR MIXTURE MODELS 12 4 Mutual information kernels for mixture models A general mixture model is de ned as P (xj ; S X s=1 s P (xjs; s ) 11) These are convenient and very powerful models for tting complicated distributions or nding cluster ....

Sam Roweis and Zoubin Gharamani. A unifying review of linear Gaussian models. Neural Computation, 11(2), 1999.

....each state. Multiple linear subspaces have previously been examined for HMMs [14, 17] Factor analysis (FA) 17] uses a di erent subspace for each component. It may be viewed as a restricted form of covariance modelling. Though ML estimation for factor analysis has simple re estimation formulae [15], the likelihood calculations are computationally expensive for LVCSR compared to other restricted covariance modelling schemes [8] This is further discussed in section 5. In [14] multiple components share the same subspace. The subspace transforms are trained in a discriminative fashion, rather ....

....and the extension to HLDA. Section 3 discusses STC systems and how they may be eciently trained. The two subspace projection schemes are then described in terms of tying the parameters of the STC system. In the following section these models are compared to the linear Gaussian models described in [15]. Finally experiments on a speaker independent task are presented. 2 Linear Discriminant Analysis This section describes a standard linear subspace projection scheme, linear discriminant analysis (LDA) Two equivalent forms of optimisation are described, the standard one based on the between to ....

[Article contains additional citation context not shown here]

S Roweiss and Z Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11:305-345, 1999.

....generative model. The more general version is described by replacing equation 1 by x( Cx( Gamma 1) w. mixture model (GMM) to model each state 2 and v is usually assumed to be generated by a GMM, which is common to all HMMs. This differs from the static linear Gaussian models presented in [7] in two important ways. First w is generated by either an HMM or GMM, rather than a simple Gaussian distribution. The second difference is that the noise is now restricted to the null space of the signal x( This type of system can be considered to have two streams. The first stream, the n 1 ....

....normalisation term is only required during recognition when multiple transforms are used. The dominant cost is a diagonal Gaussian computation for each component, O(n 1 ) per component. In contrast a scheme such as factor analysis (a covariance modelling scheme from the linear Gaussian model in [7]) has a cost of O(n 2 1 ) per component (assuming there are n 1 factors) The disadvantage of this form of generative model is that there is no simple expectation maximisation (EM) 1] scheme for estimating the model parameters. However, a simple iterative scheme is available [3] For some ....

S Roweiss and Z Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11:305--345, 1999.

....discovery [3] 1. 1 Overview We introduce a fast clustering algorithm for sparse high dimensional binary data (basket data) based on the wellknown Expectation Maximization (EM) clustering algorithm [7, 17, 6, 13] The EM algorithm is a general statistical method of maximum likelihood estimation [7, 14, 17]. In This workwas supportedby grant LM 06726from the National Library of Medicine particular it can be used to perform clustering. In our case we will use it to fit a mixture of Normal distributions to a sparse binary data set. Our algorithm is designed to efficiently handle large problem sizes ....

....suitable for very high dimensional sparse binary data. It does not incorporate sparse distance computation, regularization techniques. Initialization is done by sampling and it keeps sufficient statistics on many subsets of the data, many more than k. Also, it uses an iterative K means algorithm [14] to cluster data points in memory and then it does not make a fixed number of computations. One advantage over ours it that it only requires one scan over the data, but it makes heavier CPU use and it requires careful buffer size tuning. 6 Conclusions This paper presented a new clustering ....

Sam Roweis and Zoubin Ghahramani. A unifying review of Linear Gaussian Models. Neural Computation, 1999.

....2(b) It turns out that the maximum likelihood estimate of the factor loading matrix W in this case is given by the rst m principle eigenvectors of the sample covariance matrix, with scalings determined by the eigenvalues and sigma. Classical PCA can be obtained by taking the 0 limit [TB99, RG99] We can lift the restrictive assumption of linearity by modelling Y as a mixture of linear subspaces [TB99] This model is is shown in Figure 3(a) Q is a discrete latent (hidden) variable, that indicates which of the subspaces to use to generate Y . 2 Mathematically, this can be written as ....

S. Roweis and Z. Ghahramani. A unifying review of linear gaussian models. Neural Computation, 11(2), 1999.

No context found.

Roweis,S. and Ghahramani,Z. (1999) A unifying review of linear Gaussian models. Neural Comput., 11, 305--345.

....combining sampling methods with variational methods to estimate the quality of the variational bounds. Finally, we conclude with section 9. We assume that the reader is familiar with the basics of inference in probabilistic graphical models. For relevant tutorials he or she is referred to: [18, 12, 19, 30]. 3 Variational methods for maximum likelihood learning Variational methods have been used for approximate maximum likelihood learning in probabilistic graphical models with hidden variables. To understand their role it is instructive to derive the EM algorithm for maximum likelihood learning. ....

S. T. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305-345, 1999.

No context found.

S. Roweis, and Z. Ghahramani. "A Unifying Review of Linear Gaussian Models, " Neural Computation, 11(2):305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear gaussian models. Submitted, August 1997.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear gaussian models. Neural Computation, 11(2):305--345, 1999.

No context found.

Roweis, S. & Ghahramani, Z. (1999), `A unifying review of linear gaussian models', Neural Computation 11, 305--345.

No context found.

S. Roweis and Z. Ghahramani, "A Unifying Review of Linear Gaussian Models," Neural Computation, vol. 11, no. 2, pp. 305-345, 1999.

No context found.

S. Roweis and Z. Ghahramani, "A unifying review of linear gaussian models," Neural Comput., vol. 11, pp. 305--345, Feb. 1999.

No context found.

Sam Roweis and Zoubin Ghahramani. A unifying review of linear gaussian models. Neural Computation, 11(2):305--345, February 1999.

No context found.

S. Roweis and Z. Ghabramani. A unifying review of linear gaussian models. Neural Computation, 8, 1998. to appear.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear gaussian models. Neural Computation, 11(2):305--345, February 1999.

No context found.

Roweis, S & Ghahramani, Z (1999) A unifying review of linear gaussian models. Neural Computation 11, 305-345.

No context found.

Sam Roweis and Zoubin Ghahramani. A Unifying Review of Linear Gaussian Models. Neural Computation, 11(2), February 1998.

No context found.

S. ROWEIS AND Z. GHAHRAMANI, A unifying review of linear gaussian models. Neural Computation, vol. 11, pp. 305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani, "A Unifying Review of Linear Gaussian Models," Neural Computation, vol. 11, no. 2, pp. 305-345, 1999.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1997.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani, "A unifying review of linear Gaussian models," Neural Computation, vol. 11, no. 2, pp. 305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani, "A unifying review of linear Gaussian models," Neural Computation, vol. 11, no. 2, pp. 305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani. A Unifying Review of Linear Gaussian Models. Neural Computation, 11(2), 1999.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani, \A unifying review of linear Gaussian models", Neural Computation, vol. 11, no. 2, 1999.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of Linear Gaussian Models. Neural Computation, 1999.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani, "A unifying review of linear Gaussian models," Neural Comput., vol. 11, no. 2, pp. 306--345, 1999.

No context found.

Sam Roweis and Zoubin Ghahramani. A unifying review of linear gaussian models. Neural Computation, 11:305--345, 1999.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2):305--345, 1997.

No context found.

S. Roweis and Z. Ghahramani. A unifying review of linear Gaussian models. Neural Computation, 11(2), 1999.

No context found.

Z. Ghahramani and S. Roweis. A unifying review of linear Gaussian models. Neural Computation, 11(2):305-345, 1999.

No context found.

S.Roweis and Z.Ghahramani, "A unifying review of linear Gaussian models," Neural Computation, pp.305-345,vol.11, No.2, (1999).

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC