E. Gardner and B. Derrida. Optimal storage properties of neural network models. J. Phys. A, 21:271--284, 1988.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....digress from the teacher student scenario and take the labels S assigned to the inputs as random variables. With learning obviously impossible, a natural question to ask is: How many random patterns S can be stored correctly in a network of given architecture In the classical papers of Gardner [26, 27, 28], this question is answered for the simple perceptron. It turns out that in a large network with N adjustable weights the average number of patterns that can be stored is m = 2N ; this is equivalent to a critical load parameter # c = m N = 2. The SVM being a generalisation of the simple ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. J. Phys. A, 21:271--284, 1988.

....perceptron rule is capable of storing patterns up to the optimal capacity. Given the simplicity of single node learning it has been possible to calculate the optimal properties of single layered neural networks though for the idealised case of randomly constructed patterns with binary components [37, 50, 51, 52]. We find that for p random patterns and N input nodes it is theoretically possible to store up to p = 2N pairs of patterns in a 41 fully connected single layered neural network using the perceptron algorithm (strictly in the large N limit) By contrast the Fisher rule can only store up to p ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. Journal of Physics, A21:271--284, 1988.

....In addition, it is shown that the learning performance of Bayes optimal algorithm can be approximated by certain learning algorithms that use a neural net with a layer of hidden units to learn a perceptron. 1 Introduction Extending a line of research initiated by Elizabeth Gardner [Gar88, GD88] exceptional progress has been made in recent years in applying the methods of statistical mechanics to the analysis of the process of learning from random examples, as exemplified in the learning algorithms used to train neural networks. Recent work [DSW 87] HLW88] BH89] VJP89] LTS89] ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. J. Physics A, 21:271--284, 1988.

.... applicable (M ezard, Parisi Virasoro, 1987) This explains the success of statistical mechanical techniques such as the replica method in deriving to appear in Advances in Neural Information Processing Systems 9 (1997) the macroscopic properties of neural networks, e.g. the storage capacity (Gardner Derrida 1988), generalization ability (Watkin, Rau Biehl 1993) The replica method, though, provides much less information on the microscopic conditions of the individual dynamical variables. An alternative mean field approach is the cavity method. It is a generalization of the Thouless Anderson Palmer ....

.... by J j = 1 Gamma ff) Gamma1 1 p N X [ t ) Gamma t ] j : 3) Noting the Gaussian distribution of the cavity fields, the macroscopic properties of the neural network, such as the storage capacity, can be derived, and the results are identical to those obtained by the replica method (Gardner Derrida 1988). However, the real advantage of the cavity method lies in the microscopic information it provides. The above equations can be iterated sequentially, resulting in a general learning algorithm. Simulations confirm that the equations are satisfied in the single layer perceptron, and their ....

Gardner, E. & Derrida, B. (1988) Optimal storage properties of neural network models. Journal of Physics A: Mathematical and General 21(1):271-284.

....detectors[35] and other multilayer systems[45] Thus these approximations provide powerful theoretical tools for the study of learning from examples, at least for realizable rules. Our treatment should be contrasted with the difficult problems of the capacity of single and multilayer networks[46 48]. The capacity problems usually deal with loading random sets of data. In this case the system is highly frustrated and one has to employ the complex methods of spin glass theory, such as replica symmetry breaking[36, 49] In the learning problems of the present work, the training set consists of ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. J. Phys., A21:271--284, 1988.

....in the phase diagram of Figure 2, and is at lower ff than the zero entropy line from the annealed theory. Gardner vs. VC entropy Previous work on the statistical mechanics of learning from examples has utilized a Gibbs distribution on the space of functions, as pioneered by Elizabeth Gardner[8, 13]. In the Gardner formulation, the definition of capacity depends on whether the function class is continuous or discrete. For continuous function classes like the spherical perceptron, the Gardner entropy diverges to Gamma1 at capacity[8, 14, 15] For discrete function classes like the Ising ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. J. Phys., A21:271--284, 1988.

....Many interesting and intuitively appealing results were obtained by the physical community. Based on heuristic approaches like the replica trick, many qualitatively convincing results have been obtained. All these results are also missing from this survey. The interested reader is directed to [66, 25, 32, 31, 69, 54] for the most important of them. The reader could expect therefore that all the mathematical aspects would be presented here. She will be disappointed: only results establishing that learning process is equivalent to an information increasing (entropy decreasing) process and that the restitution ....

E Gardner, B Derrida, Optimal storage properties of neural network models, J. Phys. A: Math. Gen., 21, 271--284 (1988).

....of the simulations. We consider a Perceptron with N input neurons, all connected to a single output neuron with the weight vector w. Neuron states belong to f Gamma1; 1g. By this choice of neuron states, we can directly use theoretical results about learning and generalizing with a Perceptron [7, 11]. We use such a Perceptron to learn sets of P patterns with KLR. For each learning set Gamma, the input patterns are randomly chosen. The expected output are either randomly generated or with a Reference Perceptron. A Reference Perceptron is used to built linearly separable learning sets. It is ....

....this case ffl g is the one computed by Gyorgyi and al. We plot on figure 11 m T versus T averaged on 40 learning processes on non linearly separable learning sets. This learning sets are randomly generated. As N = 100 and P = 300, it ensures that the probability of being separable is almost zero [7]. Since the temperature is lower than 0:15, they are all detected as non separable (m T log(2) We do not encounter cases of indecision. This may be the consequence of 3 phenomena: ffl such learning sets are rare ffl approximation of the minimum by the conjugate gradient ffl discretization ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. Journal of Physics, A(21):271--284, 1988.

....network, it will evolve into the fixed point pattern under iteration of the network dynamics. Many aspects of the networks are quite well understood by now. For example, the calculation of the optimal storage capacity (number P of patterns that can be stored) was possible for a variety of models [1]. When the simple Hebb learning rule is used to set the couplings, J ij = 1 N P X =1 i j ; the network is able to store about P = 0:14 Delta N binary patterns [4] In order to improve the storage capacity and the dynamical recall properties, many different learning rules have been ....

E. Gardner and B. Derrida, Optimal storage properties of neural network models, Journal of Physics A 21, 271--284, 1988

....of vectors that can be stored, when the weight matrix and threshold vector can be chosen freely. It is known that if no requirements are set on the attraction radii of the stored vectors, then with high probability up to 2n random vectors may be stored in a network with asymmetric connections [14, 22, 83, 85]. However, when the stored vectors are not random, but are chosen in the worst possible manner, the optimal capacity is much lower. Theorem 4.1 [12, 64] The worst case capacity for Hopfield networks with at least one zero element in the diagonal of the connection matrix is 1. Proof: Clearly, at ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. Journal of Physics A: Math. Gen., 21:271--284, 1988.

....content of a real valued coupling matrix J ij . In order to compare the performance of damaged neural networks with some standard models of coding theory one needs to investigate the binary couplings (J ij = Sigma1) network. The storage capacity of this model is known to be ff cB = 0:83 [Gardner Derrida 88] Krauth M ezard 89] Krauth Opper 89] but a storage density higher than approximately ff = 0:4 cannot be reached with the currently available learning algorithms [Koehler et al. 89] The adequate model of damage in the binary couplings network is to flip synapses with a probability , that ....

E. Gardner, B. Derrida, Optimal storage properties of neural network models, Journal of Physics A 21, 271--284 (1988).

....depends on P . The greater P , the greater Diam0 . Each minimization yields a weight vector for which the norm is equal to the diameter (Figure 1) We also learn sets with random outputs for N = 100 and P = 300. These values ensure that the probability of being separable is almost zero [7]. Their non separability is effectively detected by KLR (Figure 1) We turn now to the analyze of the generalization error ffl g [11] It is the probability that a Perceptron gives the wrong output to a pattern not belonging to the learning set. In the thermodynamic limit, N 1, Opper and ....

E. Gardner and B. Derrida. Optimal storage properties of neural network models. Journal of Physics, A(21):271--284, 1988.

No context found.

B. Derrida, E. Gardner, Optimal storage properties of neural network models, J. Phys. A. 21, 271-284, 1988.

No context found.

Gardner E. and Derrida B. "optimal storage properties of neural network models". Journal of Physics A, 21:271, 1988.

No context found.

B. Derrida, E. Gardner, Optimal storage properties of neural network models, J. Phys. A. 21, 271-284, 1988.

No context found.

B. Derrida, E. Gardner, Optimal storage properties of Neural Network models, J. Phys. A. 21, 271-284, 1988.

No context found.

E. Gardner and B. Derrida. Optimal storage properties of neural network models. Journal of Physics, A(21):271--284, 1988.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC