C. P. Robert, "Mixtures of Distributions: Inference and Estimation," in Markov Chain Monte Carlo in Practice, S. Richardson W. R. Gilks and D. J. Spiegelhalter, Eds. Chapman & Hall, 1996.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....run of EM, due to the excessivenumber of components, is somewhat insensitiveto initialization. Of course we are not claiming that AEM is guaranteed to find the globally optimal mixture estimate# it is known that even MCMC mayhave difficulties escaping from local maxima of the likelihood function [24]. AEM can be used with any criterion other than MMDL, or even when k true is known: in this case, simply set k min = k true and skip the phase where the optimal k is chosen. Naturally, AEM can also be based on modified versions of EM [16] Finally, observe that the computational requirements of ....

C. Robert. Mixtures of distributions: Inference and estimation. In W. Gilks, S. Richardson, and D. Spiegelhalter, editors, Markov Chain Monte Carlo in Practice, London, 1996. Chapman & Hall.

....in section 2.1.1. Most of the updates are done using the corresponding full conditional distributions, which have the same functional forms as the corresponding priors. These full conditionals follow closely from what was published previously in [Syk00] with some modifications necessary (see e.g. [Rob96]) because we need to consider the Markov dependency between successive hidden states. As the derivations of the full conditionals do not differ much from previous work, we will omit them here and instead concentrate on an illustration how to update the latent feature space, i;s ; I i;s ) 8 i; ....

C. P. Robert. Mixtures of distributions: inference and estimation. In W. R. Gilks, S. Richardson, and D.J. Spiegelhalter, editors, Markov Chain Mont Carlo in Practice, pages 441--464. Chapman & Hall, London, 1996.

....of EM, due to the excessive number of components, is somewhat insensitive to initialization. Of course we are not claiming that AEM is guaranteed to find the globally optimal mixture estimate; it is known that even MCMC may have difficulties escaping from local maxima of the likelihood function [24]. AEM can be used with any criterion other than MMDL, or even when k true is known: in this case, simply set k min = k true and skip the phase where the optimal k is chosen. Naturally, AEM can also be based on modified versions of EM [16] Finally, observe that the computational requirements of ....

C. Robert. Mixtures of distributions: Inference and estimation. In W. Gilks, S. Richardson, and D. Spiegelhalter, editors, Markov Chain Monte Carlo in Practice, London, 1996. Chapman & Hall.

....matrix, rather than by pixel correlations. Now, it may be that this single Gaussian component will model the data adequately. If not, then we split the data into two parts and model each with a Gaussian. This can be done using the Markov Chain Monte Carlo (MCMC) sampling technique described in [10], which treats the inference as one containing hidden variables, namely the class fl i to which each datum belongs; it samples from the posteriors for the population size, means and covariances, assuming conjugate priors, whose parameters are simply those of the population as a whole. Thus the ....

C. P. Robert, "Mixtures of Distributions: Inference and Estimation," in Markov Chain Monte Carlo in Practice, S. Richardson W. R. Gilks and D. J. Spiegelhalter, Eds. Chapman & Hall, 1996.

....to the books by Titterington, Smith and Makov (1985) and Lindsay (1995) for general background. In recent years, research interest has been re generated, rst by the introduction of the Bayesian approach, with computational implementation by Markov chain Monte Carlo (MCMC) see, for example, Robert (1996) for a wide review, and secondly by allowing the number of components to vary, in this Bayesian setting as in Nobile (1994) and Richardson and Green (1997) These developments have increased the power and exibility of mixture based models, by enabling full simultaneous posterior inference about ....

Robert, C. (1996) Mixtures of distributions: inference and estimation. Chapter 24 (pp. 441-464) of Practical Markov chain Monte Carlo, W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds. Chapman and Hall, London.

....7 drawing in turn each of a set of unknown random variables or vectors, each conditionally on the values of all the other random variables. Our application of Gibbs sampling is close to applications for latent class and finite mixture models (e.g. Gelman et al. 1995, Chapter 16, and Robert, 1996). We apply this scheme to (#, #) X 1 , X n , treating (#, #) as a single random vector with the prior density f(#, #) Given current values X (p) # (p) # (p) the next values X (p 1) # (p 1) # (p 1) are determined as follows: 1. # (p 1) # (p 1) is drawn from the ....

....a model with a given number of colors, and for checking convergence of the Gibbs sampler, as indicated below. 5.2 Convergence detection The iteration steps of the Gibbs sampler are simple. Detecting convergence, however, is not straightforward. This point is discussed, among others, by Gilks and Roberts (1996) and Gelman (1996) The reason is that the process converges not to a single value but to a stationary probability distribution. We propose, for our model, a measure for improving convergence and a measure for checking convergence. The measure for improving convergence is to start the simulations ....

[Article contains additional citation context not shown here]

ROBERT, C.P. (1996), "Mixtures of distributions: inference and estimation", in Gilks et al. (1996), pp. 441 -- 464.

....that this approximation can be poor. We have thus obtained a general iid sampling method for mixture posterior distributions. This is of direct practical interest since mixtures are heavily used in statistical modelling and the corresponding inference is delicate (Titterington et al. 1985, Robert, 1996). As pointed out in the discussion of Green and Murdoch (1999) mixture models are moreover a good showcase for establishing that perfect sampling can be achieved for realistic statistical models and not only for toy problems. Green and Murdoch (1999) surmise that it might be possible to couple ....

Robert, C.P. (1996) Mixtures of distributions: inference and estimation. In W. Gilks, S. Richardson and S. Spiegelhalter, editors, Markov Chain Monte Carlo in Practice, 441-464. Chapman and Hall, London.

....is sucient prior information or knowledge about the processes producing the data, is to use a parametric model. In recent years, this has been extended through the use of Bayesian methods for model selection and the estimation of so called hyper parameters , which characterise the model, e.g. [19, 15], but while useful in particular problems, these hardly represent a general approach to the inference problem. It seems that what is needed is a method which combines the generality of kernel estimation with the power of parametric methods. This is one of the key motivations behind the MGMM ....

....of likelihood alone: it would always give preference to the more complex description. The increase in likelihood must be balanced against the cost in terms of complexity of using two components instead of one. This type of problem has received much attention in the literature on classi cation [9, 17, 19], and in system identi cation, in which model order has to be chosen [10] In both areas, widespread use has been made of Akaike s Information Criterion (AIC) which is derived from the Kullback Liebler Information Divergence between two densities: d(f; g) Z R n dxf(x) ln[ f(x) g(x) ....

[Article contains additional citation context not shown here]

C. P. Robert. Mixtures of Distributions: Inference and Estimation. In S. Richardson W. R. Gilks and D. J. Spiegelhalter, editors, Markov Chain Monte Carlo in Practice. Chapman & Hall, 1996. 15

.... distributed observations, the likelihood function L( jX) corresponding to the marginal density of equation (3) is L( jX) n Y i=1 k X j=1 j N(x i ; j ; j T j ) 4) More detailed studies of nite mixture distributions can found in such references as [16] and [13]. 4 2.3 Some Questions, Aims and Issues of Interest Other aims and issues of interest in the study of the MFA model are: Classi cation and Pattern Recognition : Predict the class (or label) of a given manifest variable, especially when some assumptions are clearly made beforehand on how many ....

....of missing variables that is proportional to the sample under study. One also has to address the inherent diculty of non identi ability of both the Factor Analysis and the mixture models. See references like [1] 3] 11] and [10] for more details on identi ability in Factor Analysis, and [16] [13] and [15] for identi ability of mixture models. We address the uncertainty about both k and q (model selection) in our subsequent research [5] 3 Estimation and Inference via Bayesian Sampling 3.1 Basic formulation and computational drawbacks In the Bayesian approach to inference and parameter ....

[Article contains additional citation context not shown here]

Robert, C. P. (1995). Mixtures of Distributions: Inference and Estimation. In W. R. Gilks, S. Richardson and D. J. Spiegelhalter (Eds.), Markov Chain Monte Carlo in Practice. Interdisciplinary Statistics, Chapter 24, pp. 441-464. Chapman and Hall.

....under certain priors one can demonstrate consistency of the posterior distribution (Roeder Wasserman 1997) there are practical di#culties since the likelihood surface, and hence the posterior distribution, is often multimodal. One consequence of multimodality, trapping states, is discussed by Robert (1996) and Mengersen Robert (1996) Markov Chain Monte Carlo (MCMC) implementations, the standard methods for examining the posterior distribution, often get trapped in areas of the parameter space where one component of the mixture has a mean placed at a data point and a small variance. In such ....

....demonstrate consistency of the posterior distribution (Roeder Wasserman 1997) there are practical di#culties since the likelihood surface, and hence the posterior distribution, is often multimodal. One consequence of multimodality, trapping states, is discussed by Robert (1996) and Mengersen Robert (1996). Markov Chain Monte Carlo (MCMC) implementations, the standard methods for examining the posterior distribution, often get trapped in areas of the parameter space where one component of the mixture has a mean placed at a data point and a small variance. In such situations, the Markov Chain ....

[Article contains additional citation context not shown here]

C. Robert (1996). Mixtures of distributions: Inference and estimation. Chapter 24 of Markov Chain Monte Carlo in Practice. W. R. Gilks, S. Richardson, and D. J. Spiegelhalter, eds. Chapman and Hall.

....one usually imposes constraints on the parameter space, e.g. by taking the location of the components to be ordered: 1 . k . This, however, is not a completely satisfactory solution, as di#erent parameter constraints could yield radically di#erent inferences (see, for example, [56], 40] 53] If, for example, a constraint of the form 1 2 is imposed, the posterior estimates of the parameters resulting from the MCMC simulation will typically produce multimodal marginal posteriors when Pr ( 1 2 y) 0. The multimodality in the estimated posteriors is especially ....

C.P. Robert, "Mixtures of distributions: inference and estimation", in Markov Chain Monte Carlo in Practice, Chapman and Hall, 1996.

....k. 4. Conclusion We have obtained what we believe to be the rst general iid sampling method for mixture posterior distributions. This is of direct practical interest since mixtures are heavily used in statistical modelling and the corresponding inference is delicate (Titterington et al. 1985, Robert, 1996). We have also illustrated that perfect sampling can be achieved for realistic statistical models and not only for toy problems. ....

Robert, C.P. (1996) Mixtures of distributions: inference and estimation. In W. Gilks, S. Richardson and S. Spiegelhalter, editors, Markov Chain Monte Carlo in Practice, 441464. Chapman and Hall, London.

....Mixtures, Rates of Convergence, Sieves. 1. INTRODUCTION Statistical inference using mixtures of Gaussians is used for many purposes including density estimation, clustering and robust estimation; see, for example, Lindsay (1995) McLachlan and Basford (1988) Banfield and Raftery (1993) and Robert (1996). When the number of components of the mixture is allowed to increase with sample size, the model is called a Gaussian mixture sieve (Grenander, 1982, Wong and Shen 1995) These sieves have been studied by several authors including Geman and Hwang (1982) Roeder (1992) Priebe (1994) and Roeder ....

Robert, C. (1996). Mixtures of distributions: inference and estimation. In: Markov Chain Monte Carlo in Practice, (W. Gilks, S. Richardson, D. Spiegelhalter, eds.) Chapman and Hall, London. p. 441- 464.

....P of observations are in group 2, 1 Gamma P in group 1. The model is thus y i Normal( T i ; T i Categorical(P ) We note that this formulation easily generalises to additional components to the mixture, although for identifiability an order constraint must be put onto the group means. Robert (1994) points out that when using this model, there is a danger that at some iteration, all the data will go into one component of the mixture, and this state will be difficult to escape from this matches our experience. Robert suggests a re parameterisation, a simplified version of which is to ....

Robert, C. (1994). Mixtures of distributions: inference and estimation. In Markov chain Monte Carlo in practice, (ed. W. Gilks, S. Richardson, and D. Spiegelhalter). Chapman and Hall. (to appear).

....number K of kernel components. In our case, this number is not given, and needs to be determined. The determination of the number of components of a mixture model is probably the most difficult step, and a completely general solution strategy is not available. Several strategies are proposed in [31, 33, 38]. However, most of the techniques are computationally expensive, and given our use of mixture models, minimizing the computational cost of the selection process becomes of paramount importance. Given a set of alternative model orders K = f1; K max g, we consider two alternative ....

C. Robert. Mixtures of distributions: Inference and estimation. In Gilks et al [22], chapter 24, pages 441--464.

....approaches to the general case where k is unknown typically adopted a different style of modelling, treating the problem as an example of Bayesian nonparametrics , and basing prior distributions on the Dirichlet process; see Escobar and West (1995) for example. Other authors, like Mengersen and Robert (1996) and Roeder and Wasserman (1995) have proposed to use, for example, a Kullback Leibler distance or a Schwarz criterion to choose the number of components. The more direct line we adopt here, of modelling the unknown k case by mixing over the fixed k case, and making fully Bayesian inference, has ....

....on the separation and the sample size. When the means are well separated, labelling of the realisations from the posterior by ordering their means will generally coincide with the population labelling; as the separation reduces, so called label switching will occur; see also Mengersen and Robert (1996). Depending on the relative separations, label switching 0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Enzyme data 3 4 5 6 7 0.0 0.2 0.4 0.6 Acidity data 10 15 20 25 30 35 0.0 0.05 0.10 0.15 0.20 Galaxy data Figure 3: Predictive densities for the 3 data sets, conditional on ....

[Article contains additional citation context not shown here]

Robert, C. (1996) Mixtures of distributions: inference and estimation, chapter 24 (pp. 441--464) of Practical Markov chain Monte Carlo, W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds. Chapman and Hall, London.

.... p(x n 1 j x n ) 1 N N X t=1 p(x n 1 j (t) 7) More details, including details of the construction of a suitable Markov chain when k is fixed, can be found in the paper by Diebolt and Robert (1994) chapters of the books by Robert (1994) and Gelman et al. 1995) and the article by Robert (1996). Richardson and Green (1997) describe the construction of a suitable Markov chain when k is allowed to vary using the reversible jump methodology developed by Green (1995) We now describe an alternative approach. 3 Constructing a Markov chain via simulation of point processes 3.1 The ....

Robert, C. P. (1996) Mixtures of distributions: Inference and estimation. In Markov Chain Monte Carlo in Practice (Eds W. R. Gilks, S. Richardson and D. J. Spiegelhalter). London: Chapman & Hall.

....to the general case where k is unknown typically adopted a different style of modelling, treating the problem as an example of Bayesian nonparametrics , and basing prior distributions on the Dirichlet process; see Escobar and West (1995) for example. Other authors, for example Mengersen and Robert (1996), Raftery (1996) and Roeder and Wasserman (1995) have proposed to use, respectively, a Kullback Leibler distance, a Laplace Metropolis estimator or a Schwarz criterion to choose the number of components. The more direct line we adopt here, of modelling the unknown k case by mixing over the fixed k ....

....on the separation and the sample size. When the means are well separated, labelling of the realisations from the posterior by ordering their means will generally coincide with the population labelling; as the separation reduces, so called label switching will occur; see also Mengersen and Robert (1996). Depending on the relative separations, label switching can be minimised by choosing to order on the variances, weights, or some combination of all three parameters. We illustrate these points in Figure 4 using a simulated data set of n = 250 points, drawn from a mixture which gives a skewed ....

[Article contains additional citation context not shown here]

Robert, C. (1996) Mixtures of distributions: inference and estimation, chapter 24 (pp. 441--464) of Practical Markov chain Monte Carlo, W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds. Chapman and Hall, London.

....Madigan and Hoeting (1993) In order to use the Gibbs sampler, I introduce the latent data z = z 1 ; z n ) where z i = j if y i N( j ; v j ) i.e. if y i belongs to the jth component of the mixture. The required conditional posterior distributions are then given by Diebolt and Robert (1994); see also the chapter by Robert (1994) The Gibbs sampler is initialized by dividing the data into J equal sized chunks of contiguous data points, using the resulting means and variances for and v, and setting ae j = 1=J (j = 1; J ) It proceeds by drawing first z, and then ae, v and ....

....to use the Gibbs sampler, I introduce the latent data z = z 1 ; z n ) where z i = j if y i N( j ; v j ) i.e. if y i belongs to the jth component of the mixture. The required conditional posterior distributions are then given by Diebolt and Robert (1994) see also the chapter by Robert (1994). The Gibbs sampler is initialized by dividing the data into J equal sized chunks of contiguous data points, using the resulting means and variances for and v, and setting ae j = 1=J (j = 1; J ) It proceeds by drawing first z, and then ae, v and in turn from their conditional ....

[Article contains additional citation context not shown here]

Robert, C. (1994). Mixtures of distributions: Inference and estimation. In Practical Markov Chain Monte Carlo (W.R. Gilks, D.J. Spiegelhalter and S. Richardson, eds.), to appear.

....p( j x 1 ; x 100 ) Convergence of the chain in this case was judged by eye by looking at graphs such as those shown in Figure 1. Generally choosing a suitable starting value and determining convergence of the chain are difficult problems, with which we do not concern ourselves here. See Robert (1995, 1996), Cowles and Carlin (1996) and references therein for further information. ....

Robert, C. P. (1996) Mixtures of distributions: Inference and estimation. In Markov Chain Monte Carlo in Practice (Eds W. R. Gilks, S. Richardson and D. J. Spiegelhalter) . London: Chapman & Hall.

No context found.

C. P. Robert, "Mixtures of Distributions: Inference and Estimation," in Markov Chain Monte Carlo in Practice, S. Richardson W. R. Gilks and D. J. Spiegelhalter, Eds. Chapman & Hall, 1996.

No context found.

Robert C.P. (1996), Mixtures of distributions: Inference and estimation. In Markov Chain Monte Carlo in Practice (Eds W.R.Gilks, S.Richardson and D.J.Spiegelhalter). London: Chapman & Hall.

No context found.

Robert, C. (1996) Mixtures of distributions: inference and estimation. Chapter 24 (pp. 441--464) of Practical Markov chain Monte Carlo, W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds. Chapman and Hall, London.

No context found.

Robert, C.P. (1996): Mixtures of distributions: inference and estimation, in Gilks, Richardson, Spiegelhalter (eds.), Markov Chain Monte Carlo in Practice, Chapman & Hall, London, pp. 441--464.

No context found.

Robert, C. (1996) Mixtures of distributions: inference and estimation. Chapter 24 (pp. 441--464) of Practical Markov chain Monte Carlo, W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds. Chapman and Hall, London.

No context found.

Robert, C. (1996) Mixtures of distributions: inference and estimation. Chapter 24 (pp. 441--464) of Practical Markov chain Monte Carlo, W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds. Chapman and Hall, London.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC