E. B. Baum and K. E. Lang, \Constructing hidden units using examples and queries," in Advances in Neural Information Processing Systems 3 (R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds.), pp. 904-910, San Mateo: Morgan Kaufmann, 1991.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....runs for 200 iterations, it obtains the same number of codewords found after 15 iterations. Afterwards, we have executed the testing of the results obtained by using the same test set employed by Fritzke, constituted by 576 points. He compares this result with the one obtained by Baum and Lang [53] that, for their best model, report an average of 29 errors on the test set. Fritzke, with 145 codewords, obtains zero errors on the same test set; FACS too achieves this result with the 144 codewords of Fig. 17. Number of iterations. In [32] Fritzke underlines that, for every learning method, ....

E. B. Baum and K. E. Lang, \Constructing hidden units using examples and queries," in Advances in Neural Information Processing Systems 3 (R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds.), pp. 904-910, San Mateo: Morgan Kaufmann, 1991.

....runs for 200 iterations, it obtains the same number of codewords found after 15 iterations. Afterwards, we have executed the testing of the results obtained by using the same test set employed by Fritzke, constituted by 576 points. He compares this result with the one obtained by Baum and Lang [92] that, for their best model, report an average of 29 errors on the test set. Fritzke, with 145 codewords, obtains zero errors on the same test set; FACS too achieves this result with the 144 codewords of Fig. 6.13. 1 0.8 0.6 0.4 0.2 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 ....

E. B. Baum and K. E. Lang, \Constructing hidden units using examples and queries," in Advances in Neural Information Processing Systems 3 (R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds.), pp. 904-910, San Mateo: Morgan Kaufmann, 1991.

....network exactly classifies its inputs according to P , but the learning algorithm will have probability at least 0.01 of finding a choice of weights yielding an error greater than . 65; 293 p 1 Gamma 2:7 10 Gamma8 142; 121 p 1 Gamma 2:9 10 Gamma96 Heuristic [Baum, 1990, Baum and Lang, 1991] If a data base of examples is loaded onto a jW j weight net (for AE jW j) one expects to make a fraction jW j errors in classifying future examples drawn from the same distribution. ANNIE Tutorial Theories and heuristics for MLP St Louis, Nov 13, 1994 38 Capacity of a ....

Baum, E. and Lang, K. (1991). Constructing hidden units using examples and queries. In Lippman, R., Moody, J., and Touretzky, D., editors, Neural Information Processing 3, pages 904--910. Morgan Kaufmann, San Mateo, CA.

....density of data points (which is higher in the center of the spirals) makes it also a challenge for networks of local units. As for most learning problems the interesting aspect is not learning the training examples but rather the performance on new data which is often denoted as generalization. Baum Lang (1991) defined a test set of 576 points for this problem consisting of three equidistant test points between each pair of adjacent same class training points. They reported for their best network 29 errors on the test set in the mean. In figure 4 a typical network generated by our method can be seen as ....

Baum, E. B. & K. E. Lang [1991], "Constructing hidden units using examples and queries," in Advances in Neural Information Processing Systems 3, R.P. Lippmann, J.E. Moody & D.S. Touretzky, eds., Morgan Kaufmann Publishers, San Mateo, 904-- 910.

....for dynamically adding neurons to the initial knowledge based network. Their approach starts with an initial network representing the domain theory and modifies this theory by constructing a single hidden layer of threshold logic units (TLUs) from the labeled training data using the HDE algorithm [1]. The HDE algorithm divides the feature space with hyperplanes. Fletcher and Obradovi c s algorithm maps these hyperplanes to a set of TLUs and then trains the output neuron using the pocket al..gorithm [8] The KBDistAl algorithm proposed in this paper, like that of Fletcher and Obradovi c, also ....

Baum, E., and Lang, K. 1991. Constructing hidden units using examples and queries. In Lippmann, R.; Moody, J.; and Touretzky, D., eds., Advances in Neural Information Processing Systems, vol. 3, 904--910. San Mateo, CA: Morgan Kaufmann.

....for dynamically adding neurons to the initial knowledge based network. Their approach starts with an initial network representing the domain theory and modifies this theory by constructing a single hidden layer of threshold logic units (TLUs) from the labeled training data using the HDE algorithm [20]. The HDE algorithm divides the feature space with hyperplanes. Fletcher and Obradovi c s algorithm maps these hyperplanes to a set of TLUs and then trains the output neuron using the pocket al..gorithm [21] The RAPTURE system is designed to refine domain theories that contains probabilistic rules ....

E. Baum and K. Lang, "Constructing hidden units using examples and queries," in Advances in Neural Information Processing Systems, vol. 3, R. Lippmann, J. Moody, and D. Touretzky, Eds., San Mateo, CA, 1991, pp. 904--910, Morgan Kaufmann.

....Figure III.21. Tche de la double spirale Pourquoi choisir cette tche pour tester l apprentissage progressif# Essentiellement parce qu il s agit d une tche trs difficile rsoudre pour un rseau couches non incrmental#: un rseau rtropropagation du gradient ne parvient pas apprendre les exemples [Baum Lang 91] Fahlman Lebiere 91]#; des expriences utilisant des variantes de la rtropropagation parviennent un rsultat, en un nombre de prsentations trs variable, selon les variantes (minimum 7600) Dans ces expriences, le taux de russite n est pas mentionn#; autrement dit, on s est content de montrer un ....

.... seul essai comportant 100000 prsentations ncessite dj plus de cinq milliards d adaptations de poids# Les rsultats sont trs nets#: quand on fait apprendre l ensemble entier (apprentissage global) 4 essais convergent vers une solution, 21 chouent, ce qui confirme les essais 221 infructueux de [Baum Lang 91]#; dans le cas o l apprentissage est progressif, 20 essais convergent, 5 chouent. L apprentissage progressif permet donc un rseau donn, dot d un algorithme d apprentissage donn, de rsoudre une tche qui est insoluble quand l ensemble d apprentissage est prsent en entier (pour une initialisation ....

Eric B. Baum & Kevin J. Lang. Constructing Hidden Units using Examples and Queries. In R.P. Lippmann et al. (Eds) Advanced in Neural Information Processing 3, San Mateo, California: Morgan Kaufman, p. 904-910, 1991.

....10 3 Gamma 10 5 , still resulting to a prohibitively long learning time. Another approach to improve the convergence is based on the idea to incorporate a priori knowledge by a proper initialization of weights. Many researchers have reported a significant improvement of the convergence [4], 16] 10] 17] against commonly used random initialization technique. Yet another approach is an attempt to avoid high dimensional search in the whole space of parameters by exploiting a partial, iterative optimization approach when only a part of weights is trained while the rest part of the ....

E. B. Baum and K. J. Lang. Constructing hidden units using examples and queries. In R. P. Lippman, S. J. Hanson, and D. S. Touretzky, editors, Advances in Neural Information Processing Systems, volume 3, pages 904--910. Morgan Kaufmann, San Mateo, CA, 1991.

....They used a variant of the quick prop learning algorithm [10] with weight decay. They claimed that the problem could not be solved with simpler architecture (i.e. less layers or without short cuts) Their result on the same data set seems to give poor generalization results. Baum and Lang [1] demonstrated that there are many sets of weights that would cause a 2 Gamma 50 Gamma 1 network to be consistent with the training set, however, the single layer feed forward architecture trained with error back propagation was unable to find any of them when starting with random initial ....

E. Baum and K. Lang. Constructing hidden units using examples and queries. In R. P. Lippmann, J. E. Moody, and D. S. Touretzky, editors, Advances in Neural Information Processing Systems, volume 3, pages 904--910. Morgan Kaufmann, San Mateo, CA, 1991.

....how some concept classes that are dense in themselves can be learned efficiently if we allow the learner access to random unlabeled examples. This added capability enables the learner to maintain its sensitivity to the input distribution, while reducing the number of labels that it needs to know. Baum [Baum, 1991], proposed a learning algorithm that uses membership queries to avoid the intractability of learning neural networks with hidden units. His algorithm is proved to work for networks with at most 4 hidden units, and there is experimental evidence [Baum and Lang, 1991] that it works for larger ....

....of labels that it needs to know. Baum [Baum, 1991] proposed a learning algorithm that uses membership queries to avoid the intractability of learning neural networks with hidden units. His algorithm is proved to work for networks with at most 4 hidden units, and there is experimental evidence [Baum and Lang, 1991] that it works for larger networks. However, when Baum and Lang tried to use this algorithm to train a network for classifying handwritten characters, they encountered an unexpected problem [Baum and Lang, 1992] The problem was that many of the images generated by the algorithm as queries did not ....

E. B. Baum and K. Lang. Constructing hidden units using examples and queries. In Advances in Neural Information Processing, volume 3, 1991.

....of the other class. The resulting cuts in the spiral can be interpreted as poor generalization. In absence of other evidence it seems more natural to assume that those intermediate points belong to the same class. The decision regions produced by the network of Lang and Witbrock look similar. Baum Lang (1991) proposed a constructive method and also tested it with the twospiral problem. Their approach employs an oracle that can tell for every point in the plane the desired class. Queries to the oracle are then used to position the hyperplanes Initialize cell structure A with one k dimensional ....

Baum, E. B. & K. E. Lang (1991), Constructing hidden units using examples and queries, in advances in neural information processing systems 3 , R.P. Lippmann, J.E. Moody & D.S. Touretzky, eds., Morgan Kaufmann Publ., Inc, San Mateo, pp. 904--910.

....Section 5 considers the computational costs of these experiments, and Section 6 concludes with a discussion of the results and implications for future work. 1 In some cases active selection of training data can sharply reduce worst case computational complexity from NP complete to polynomial time [Baum and Lang, 1991], and in special cases to linear time. Learning algorithm novel input Training set Final network network weights predicted output Passive learning Learning algorithm new output novel input Training set Final network network weights predicted output Active learning Environment new input to try ....

.... Average case analysis indicates that on many domains the expected performance of active selection of training examples is significantly better than that of random sampling [Freund and Seung, 1993] these results have also been supported by empirical studies [Cohn et al. 1990; Hwang et al. 1991; Baum and Lang, 1991]. A limitation of the active learning algorithms mentioned above is that they are only applicable to specific active learning problems: the algorithms of Cohn et al. and Hwang et al. are limited to classification problems, and Baum and Lang s algorithm is further limited to a 3 Consider cases ....

E. Baum and K. Lang. (1991) Constructing hidden units using examples and queries. In R. Lippmann et al., eds., Advances in Neural Information Processing Systems 3, Morgan Kaufmann, San Francisco, CA.

....TopGen and Regent, Fletcher and Obradovic (1993) present an approach that adds nodes to a Kbann network. Their system constructs a single layer of nodes, fully connected between the input and output nodes, off to the side of the Kbann network. They generate new hidden nodes using a variant of Baum and Lang s (1991) constructive Connectionist Theory Refinement algorithm. Baum and Lang s algorithm first divides the feature space with hyperplanes. They find each hyperplane by randomly selecting two points from different classes, then localizing a suitable split between these points. Baum and Lang repeat this ....

Baum, E., & Lang, K. (1991). Constructing hidden units using examples and queries. In Lippmann, R., Moody, J., & Touretzky, D. (Eds.), Advances in Neural Information Processing Systems, Vol. 3, pp. 904--910, San Mateo, CA. Morgan Kaufmann.

....the input and output units, off to the side of Kbann, in a style similar to Strawman. Their approach differs from Strawman mainly in the training of the network, as well as the fact that only one network is considered during their search. Their new hidden units are determined using a variant of Baum and Lang s (1991) constructive algorithm. Baum and Lang s algorithm first divides the problem domain space with hyperplanes, thereby determining the number of new hidden units to be added. Then the weights between the new hidden units and the inputs units are determined by these hyperplanes. Finally, the weights ....

Baum, E. & Lang, K. (1991). Constructing hidden units using examples and queries. In Lippmann, R., Moody, J., & Touretzky, D., editors, Advances in Neural Information Processing Systems (volume 3), (pp. 904--910), San Mateo, CA. Morgan Kaufmann.

....is uncertain. This approach shows promise for concept learning in cases with relatively compact, connected concepts, and has already produced impressive results on the power system static security problem. It is, however, susceptible to the pathology discussed in Section 3.1. An algorithm due to Baum and Lang (1991), uses queries to reduce the computational costs of training a single hidden layer neural network. Their algorithm makes queries that allow the network to efficiently determine the connection weights from the input layer to the hidden layer. Seung et al. 1992) independently proposed a similar ....

E. Baum and K. Lang. (1991) Constructing hidden units using examples and queries. In R. Lippmann et al., eds., Advances in Neural Information Processing Systems 3, Morgan Kaufmann.

.... modified theory) Fletcher and Obradovi c s algorithm starts with an initial network representing the domain theory and modifies this theory by training a single hidden layer of threshold neurons using the labeled training data [11] It uses the hyperplane detection from examples (HDE) algorithm [30] to construct the hidden layer. Each hidden neuron corresponds to a hyperplane. Fletcher and Obradovi c s algorithm maps these hyperplanes to a set of threshold neurons and then then trains the output neuron using the pocket al..gorithm [4] Our approach is similar to the one taken by Fletcher and ....

E. Baum and K. Lang, "Constructing hidden units using examples and queries," in Advances in Neural Information Processing Systems, vol. 3, R. Lippmann, J. Moody, and D. Touretzky, Eds., San Mateo, CA, 1991, pp. 904--910, Morgan Kaufmann.

.... networks 2 X 2OR [Alpaydin 90.1] Grow and Learn 3 N N 1O I [Diederich 88] Neuron Recruitment 4 N [Reilly 82] Category Learning 3 N X 0 I [Ivakhnenko 68] GMDH 1 x L N X 1 I [Barron 75] GMDH McLaurin series 1 x L N X 1 [Tenorio 89] GMDH Simulated Annealing 1 x L N X 1 [Baum 91] MLP Query Learning 3 X 2FR [Fiesler 92.2] HONN 9 Superceptron 2 W 0 [Ring 93.2] HONN 2 x L 2FR 1 Backpropagation 2 Multilayer Perceptron 3 Stochastic Delta Rule 4 transform = cluster of neurons 5 NEural Units Recruitment ALgorithms 6 combination of ....

E. B. Baum and K. J. Lang. Constructing hidden units using examples and queries. In R. P. Lippmann et al., editor, NIPS 3. Morgan Kaufmann, 1991.

....unless we have some indication of where this important part is, we may have no chance of locating it. To obviate this problem, most work on such 11 domains, including the work described later in this dissertation, uses the paradigm of learning from both membership queries and random examples [BL91, Eis91] 1.3 Two Learning Systems In this section, we discuss two learning systems with which we will be concerned in the following chapters: feedforward neural networks, and vector quantizers. Although they have many other applications, we will only be concerned with neural networks as binary ....

....on this approach reported in [Yam91] selects training examples based on the sensitivity of a partially trained network to perturbations of the pattern. This combines the advantage of using distribution information with the practicality demonstrated in [HCOM90] An algorithm due to Baum and Lang [BL91] uses queries to reduce the computational costs of training a single hidden layer neural network. Their algorithm makes queries that allow the network to efficiently determine the connection weights from the input layer to the hidden layer. Recent work by David MacKay [Mac91] pursues a logical ....

E. Baum and K. Lang. Constructing hidden units using examples and queries. In R. Lippmann, J. Moody, and D. Touretzky, editors, Advances in Neural Information Processing Systems 3, San Jose, 1991. Morgan Kaufmann.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC