Citation Context

...ator can be unacceptably slow. In fact, it is widely believed that learning variational parameters using gradient estimators of the form (6) is infeasible (Hinton & Zemel, 1994; Dayan & Hinton, 1996; =-=Kingma & Welling, 2013-=-). In the next section we will show how to make this approach practical by applying variance reduction techniques. 2.3. Variance reduction techniques Though gradient estimates computed using Eq. 6 are...

Citation Context

... et al., 2012) analyzed the elements of the Hessian w.r.t. the parameters in neural network context. The differentiable reparameterization of latent variables in this paper was introduced earlier in (=-=Kingma & Welling, 2013-=-) and independently in (Bengio, 2013), but these publications lacked a theoretic analysis of the impact on the efficiency of inference. In (Kingma & Welling, 2013), the reparameterization trick was us...

Citation Context

...hm as an instance of population VI, thus reinterpreting it as minimizing the KL divergence to the population posterior. We recover SVI by setting α equal to the number of data points in the data set and replacing the stream of data F with Fx, the empirical distribution of the observations. The “stream” in this case comes from sampling with replacement from Fx, which results in precisely the original SVI algorithm.2 We focused on the conditionally conjugate family for convenience, i.e., the simple gradient in Eq. 8. We emphasize, however, that by using recent tools for nonconjugate inference [17, 18, 24], we can adapt the new ideas described above—the population posterior and the F-ELBO—outside of conditionally conjugate models. Finally, we analyze the population posterior distribution under the assumption the only way the stream affects the model is through the data. Formally, this means the unobserved variables in the model and the stream Fα are independent given the data X. The population posterior without the local latent variables z (which can be marginalized out) is EFα [p(β |X)]. Expanding the expectation gives ∫ p(β |X)p(X |Fα)dX, showing that the population posterior distribution can...

Citation Context

...he traditional variational inference framework [7, 1] by incorporating stochastic approximation [16] into the optimization of the variational lower bound. Currently, there exist two major research directions in stochastic variational inference. The first one (data stochasticity) attempts to deal with massive datasets by constructing stochastic gradients by using mini-batches of training examples [5, 6]. The second direction (expectation stochasticity) aims at dealing with the intractable expectations under the variational distribution that are encountered in non-conjugate probabilistic models [12, 14, 10, 18, 8, 15, 20]. Unifying these two ideas, it is possible to use stochastic gradients to address both massive datasets and intractable expectations. This results in a doubly stochastic estimation approach, where the mini-batch source of stochasticity can be combined with the stochasticity associated with sampling from the variational distribution. In this paper, we are interested to further investigate the expectation stochasticity that in practice is dealt with by drawing samples from the variational distribution. A challenging issue here is concerned with the variance reduction of the stochastic gradients....

Citation Context

...he direction of the gradient of the ELBO with respect to those hyperparameters. We can also extend the framework developed in this paper to models with hierarchies of global parameters as in appendix A of (Hoffman et al., 2013). 3 Related Work The idea of sampling from global variational distributions to optimize intractable variational inference problems has been proposed previously in several contexts. Ji et al. (2010); Nott et al. (2012); Gerrish (2013); Paisley et al. (2012), and Ranganath et al. (2014) proposed sampling without a change of variables as a way of coping with non-conjugacy. Kingma and Welling (2014) and Titsias and Lazaro-Gredilla (2014) proposed methods that do use a change of variables, although their methods focus more on speed and/or dealing with nonconjugacy than on improving the accuracy of the variational approximation. Salimans and Knowles (2013) also used a change of variables to dramatically reduce the variance of their stochastic gradients. Although some of the above methods use stochastic optimization to improve the quality of the mean-field approximation, there are major differences between these methods and SSVI. Ji et al. (2010) apply their method to models where some par...

Citation Context

...ess relevant when a lot of labeled data are available. Nonetheless, there are a few recent research attempts to revive this area, for example, using variational methods for probabilistic autoencoders =-=[24]-=-. 3 Convolutional neural networks Since 2012, one of the most important results in Deep Learning is the use of convolutional neural networks to obtain a remarkable improvement in object recognition fo...

Citation Context

...obabilistic models. With these models the key issue concerns the intractability of inference in the latent variables, e.g., Helmholtz Machines (Dayan et al., 1995) and variational autoencoders (VAE) (=-=Kingma & Welling, 2013-=-). The encoder is used to approximate a posterior distribution and the decoder is used to stochastically reconstruct the data from latent variables. Gregor et al. (2015) further introduced the Deep Re...