HINTON, G. E., AND SEJNOWSKI, T. J. 1986. Learning and relearning in Boltzmann machines. In Parallel Distributed Processing, Volume 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. 282--317.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... so badly on this problem One hypothesis is that BP relies on low order statistical information extracted from the training set [Thornton,Measuring,Forthcoming] The poor generalization is then explained, since parity problems are known to exhibit no low order statistical regularities whatsoever [2]. This result is partially demonstrated by Table 1, which plots out all the conditional output probabilities for first order input cases (i.e. single, input variable instantiations) Note that all the probabilities are at their chance level of 0.5. C P(C) P(y1=1 C) P(y1=0 C) 1 0.5 0.5 x4=1 ....

Hinton, G. and Sejnowski, T. (1986). Learning and relearning in boltzmann machines. In D. Rumelhart, J. McClelland and the PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition. Vols I and II (pp. 282-317). Cambridge, Mass.: MIT Press.

....on all the connections from all input units except input 6 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 epoch pdp:testset mean error #1 8 4 1 pdp:mean error #1 8 4 1 Figure 2: unit number 1. However, when we come to examine the Hinton diagram [7] for non input nodes in the the network (see Figure 3) we certainly do not see this effect. In fact what we see is that the learning has produced quite pronounced positive and negative weightings for connections from higher numbered input units. 7 13 9 10 11 12 In a sense, this is only to be ....

Hinton, G. and Sejnowski, T. (1986). Learning and relearning in boltzmann machines. In D. Rumelhart, J. McClelland and the PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition. Vols I and II (pp. 282-317). Cambridge, Mass.: MIT Press. 9

....capturing of relations. As has been known for some time [16] problems which involve relational effects are, in general, harder to solve than problems which do not. Another, reliable heuristic approach involves determining the order of statistical effect which underpins the solution to the problem [17]. The idea here is that higher order effects are harder to exploit. Are these heuristics unfounded If so, we need to ask why researchers find them relatively reliable. If they are not unfounded then we need to determine precisely what their true foundation is. If the foundation turns out to be ....

Hinton, G. and Sejnowski, T. (1986). Learning and relearning in boltzmann machines. In D. Rumelhart, J. McClelland and the PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition. Vols I and II (pp. 282-317). Cambridge, Mass.: MIT Press.

....agent who makes use of constructive processes. These processes are, I will argue, essentially redescriptive in character and therefore provide a good, low level model for the RRH part one. 3 Statistical cases Statistical theory tells us that any data set can exhibit properties of various types. [7] A basic distinction is between type 1 (type 1) and type 2 (higherorder) properties. The type 1 (statistical) properties of a particular data set are the relative frequencies (i.e. probabilities) of variable values. A type 1 property of a set of instantiation vectors might be the fact that the ....

Hinton, G. and Sejnowski, T. (1986). Learning and relearning in boltzmann machines. In D. Rumelhart, J. McClelland and the PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition. Vols I and II (pp. 282-317). Cambridge, Mass.: MIT Press.

....between these networks and conventional three layer feed forward networks. Evidently, the information capacity results apply to the more conventional feed forward network as well. The network model presented here bears some resemblance to models involving hidden (or latent) variables (see e.g. [7]) however, there is one important difference: namely, the hidden variables in other models are only hidden in the sense that they are isolated from the network s inputs and outputs; but they are not isolated from each other, they are allowed full participation in the dynamics, including direct ....

HINTON, G. E., AND SEJNOWSKI, T. J. Learning and Relearning in Boltzmann Machines. In Rumelhart et al. [21], 1986, pp. 282--317.

.... and have focused on the estimation of static quantities such as stimulus location [Anderson and Van Essen, 1994, Zemel et al. 1998, Deneve et al. 1999, Pouget et al. 2000] Other models have relied on mean field approximations or various forms of Gibbs sampling for perceptual inference [Hinton and Sejnowski, 1986, Dayan et al. 1995, Dayan and Hinton, 1996, Hinton and Ghahramani, 1997, Rao and Ballard, 1997, Rao, 1999, Rao and Ballard, 1999, Hinton and Brown, 2002] We describe a new approach to Bayesian computation in a cortical network model. We specify how the feedforward and recurrent connections in ....

Hinton, G. and Sejnowski, T. (1986). Learning and relearning in Boltzmann machines. In Rumelhart, D. and McClelland, J., editors, Parallel Distributed Processing, volume 1, chapter 7, pages 282--317. MIT Press, Cambridge.

....completing # to the best fitting # . The weights # are assumed to be subject to slow plastic changes in order to fit distributions such as that in equation 1 to the statistics of the patterns presented. The learning rule is based on the standard Boltzmann Machine learning algorithm [9] using Gibbs sampling, with the modification that, in the negative phase, only one full step of Gibbs sampling is done [10] This learning rule involves one phase of Hebbian learning driven by activity patterns from the world, and one phase of anti Hebbian learning driven by patterns generated in ....

Hinton, G. and Sejnowski, T.J. (1986). Learning and relearning in Boltzmann machines. In Rumelhart, D.E. and McClelland, J.L. (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations. Cambridge, MA, MIT Press.

....framework, the environment generates training examples which we will refer to as observations by sampling from one distribution; the other distribution is embodied in the model. Examples of generative frameworks are mixtures of Gaussians (MoG) 2] factor analysis [4] and Boltzmann machines [8]. In the recoding unsupervised framework, the model transforms points from an obser vation space to an output space, and the output distribution is compared either to a reference distribution or to a distribution derived from the output distribution. An example is independent component analysis ....

G. E. Hinton and T. J. Sejnowski. Learning and relearning in Boltzmann machines. In Parallel Distributed Processing, volume 1, pages 282-317. MIT Press, 1986.

....by network topology, node characteristics, and training rules. Frequently used and discussed models are the multilayer feedforward networks with supervised learning by error backpropagation [3] and the feedback networks, either with symmetric connectivity and stochastic nodes (Boltzmann machines [4, 5]) symmetric connectivity and deterministic nodes (Hopfield net [6, 7, 8] or nonsymmetric connectivity and deterministic nodes[9, 10] In order to be as general as possible in the implementation studies we use a feedback algorithm without any assumption on symmetry of the weight matrix. Thus, ....

Hinton, G. E. and T. J. Sejnowski. "Learning and relearning in Boltzmann machines." In vol. II of [3].

....the overall priors on the model parameters, how do we choose priors for these parameters In this chapter I use zero mean Gaussian priors on each distinct parameter vector w and v . Gaussian priors correspond to the traditional method of ridge regression [90] in statistics or weight decay [86] in neural networks. For expert parameter vector w the prior is ) 0 21 ) 1 (7.2) and for gate parameter vector v the prior is given by: 0 1 4 (7.3) The hyper parameters represent the reciprocal of the variance of these Gaussian ....

Hinton, G. E. and Sejnowski, T. J. [1986], Learning and relearning in Boltzmann machines, in D. E. Rumelhart and J. E. McClelland, eds, `Parallel Distributed Processing', MIT Press, Cambridge Mass., pp. 282--317.

....a test that determines whether the number of ones or zeros in an array of binary digits is odd or even [8] For example, in parity problems, the rule for the true cases is follows: The output is true if an odd number of ones are encountered among the n number of inputs. It has been shown in [3] that parity mappings do not show any regularity which can be described in the form of a direct correlation between value(s) of particular inputs and value(s) of output. So, parity problems are well known examples of relational problems. A general attitude in solving parity problems is to provide ....

G. Hinton and T. Sejnowski. Learning and re-learning in boltzmann machines. In D. Rumelhart, J. McClelland, and the PDP Research Group, editors, Parallel Distributed Processing: Explorations in the Microstructures of Cognition. Vols I and II. MIT Press, Cambridge, Mass., 1986.

.... this alternating Gibbs sampling is run to equilibrium, there is a very simple way to update the weights so as to minimize the Kullback Leibler divergence, Q , between the data distribution, Q , and the equilibrium distribution of fantasies over the visible units, Q , produced by the RBM [4]: 0 s i s j Q1 (3) where s i s j Q 0 is the expected value of s i s j when data is clamped on the visible units and the hidden states are sampled from their conditional distribution given the data, and s i s j Q1 is the expected value of s i s j after prolonged Gibbs sampling. This ....

G. E. Hinton and T. J. Sejnowski. Learning and relearning in boltzmann machines. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations. MIT Press, 1986.

No context found.

HINTON, G. E., AND SEJNOWSKI, T. J. 1986. Learning and relearning in Boltzmann machines. In Parallel Distributed Processing, Volume 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. 282--317.

No context found.

Hinton, G. E. & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Volume 1, 282-317. Cambridge, MA: MIT Press.

No context found.

G.E. Hinton and T.J. Sejnowski. Learning and relearning in Boltzmann machines. In Rumelhart and McClelland [3], pages 282--317.

No context found.

Hinton, G. E. & Sejnowski, T. J. Learning and Relearning in Boltzmann Machines. In J. L. McClelland, D. E. Rumelhart, and the PDP Research Group, Parallel Distributed Processing, vol. 1. The MIT Press, Cambridge, Massachusetts, 1986.

No context found.

Hinton, G. E. and Sejnowski, T. J. 1986. Learning and relearning in Boltzmann machines. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Rumelhart D.E., McClelland J. L., and the PDP Research Group, editors. Cambridge, MA: MIT Press, v.1, ch. 7.

No context found.

G.E. Hinton and T.J. Sejnowski. Learning and relearning in Boltzmann machines. In Rumelhart and McClelland [3], pages 282--317.

No context found.

G.E. Hinton and T.J. Sejnowski. Learning and relearning in boltzmann machines. In McCelland Rumelhart, editor, Parallel Distributed Processing, volume 1. MIT Press, 1986.

No context found.

G.E. Hinton and T.J. Sejnowski. Learning and relearning in Boltzmann machines. In D.E. Rumelhart, J.L. McClelland, and the PDP Research Group, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Vol. 1: Fundations, pages 283--317. MIT Press, 1986.

No context found.

G. E. Hinton and T. J. Sejnowski. Learning and relearning in Boltzmann machines. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume I, pages 282--317. MIT Press, Cambridge MA., 1986.

No context found.

G. E. Hinton and T. J. Sejnowski, "Learning and relearning in Boltzmann machines," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition (D. E. Rumelhart and J. L. McClelland, eds.), vol. I, pp. 282--317, Cambridge MA.: MIT Press, 1986.

No context found.

G. E. Hinton and T. J. Sejnowski, "Learning and relearning in Boltzmann machines," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, pp. 282--317.

No context found.

Hinton, GE& Sejnowski, TJ (1986). Learning and relearning in Boltzmann machines. In DE Rumelhart, JL McClelland and the PDP research group, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, Cambridge, MA: MIT Press, 282-317.

No context found.

Hinton, G.E. and Sejnowski, T.J., "Learning and relearning in Boltzmann machines," in Parallel Distributed Processing, (MIT Press, Cambridge, MA, 1986), ch. 7.

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC