C. Goller and A. Kuchler. Learning task-dependent distributed structurerepresentations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347-352, 1996.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... and learning structured (or hierarchically organized) information begun in the early 90 s with recursive auto associative memories (RAAM) 31] Since then, several other architectures have been proposed, including holographic reduced representations (HRR) 29] and recursive neural networks (RNN) [15, 35, 9]. A selection of papers in this research area recently appeared in [10] RNN are capable of solving the supervised learning problems such as classification and regression when the output prediction is conditioned on a hierarchical data structure, like the structural representation of fingerprints ....

....model, the log likelihood has the form l(D; #) t log Y c t (5) where # denotes the set of trainable weights and t ranges over training examples. Optimization is performed with a gradient descent procedure, where gradients are computed by the backpropagation through structure algorithm [15, 35]. We remark that the state vector X(s) at the supersource is a distributed representation of the entire input DPAG and encodes features of the input DPAG deemed to be relevant for discrimination amongst classes. In the subsequent experiments, the components of the state vector at the supersource ....

C. Goller and A. Kuchler. Learning task-dependent distributed structure-representations by backpropagation through structure. In IEEE Int. Conf. on Neural Networks, pages 347--352, 1996.

....is shown. On the right, the state transition network is unrolled bottom up (following solid lines) to match the topology of the input graph. Learning is accomplished by a special form of backpropagation referred to as backpropagation through structure (BPTS) The algorithm was first proposed in [11] for the limited case of labeled ordered m ary trees. To explain the algorithmic idea, first note that all the copies of transition and output networks linked according to the edge set of the input graph can be thought of as a global feedforward neural network on which gradients can be computed ....

C. Goller and A. Kuchler. Learning task-dependent distributed structure-representations by backpropagation through structure. IEEE Trans. on Neural Networks, pages 34752, 1996.

....i.e. which depend on the actual clause occurrence at run time, like the current total number of uses of a given clause in the current search or the number of variables of calling clause becoming instantiated. 9 The results obtained with this network and other standard neuralnetwork models [47] improved the search time of one order of magnitude, however, as is clear from the discussion in Section V A and the examples of features given above, the encoding of structural information in a fixed size vector is not able to capture all the relevant information gathered by 9 Note that this ....

....A preliminary move toward this direction is reported in [48] where LRAAM based networks were successfully used to perform classification of symbolic recursive structures encoding logical terms. A refinement of this work led to the definition of the backpropagation through structure algorithm [47]. The experimental comparison between the LRAAM based approach and the backpropagation through structure algorithm reported in [47] shows that the latter algorithm obtains slightly better results for all examples solvable by the LRAAM based approach, but with smaller networks and training times. ....

[Article contains additional citation context not shown here]

C. Goller and A. Kuchler, "Learning task-dependent distributed structure-representations by backpropagation through structure," in IEEE Int. Conf. Neural Networks, 1996, pp. 347--352.

....tree is the correct one (but which one is unknown) we must have in this case m i # i=1 y ij = 1. 3) The structured nature of the objects we are dealing with (syntactic trees) suggests adopting a recently introduced class of connectionist architectures, called recursive neural networks (RNNs) [2, 1]. Unlike feedforward neural networks, which are grounded on attribute value representations of data, RNNs can adaptively process labeled graphs, e#ectively taking advantage of relations among atomic entities that constitute a single instance. Here we employ a novel and specialized version of these ....

.... set, the second sum ranges over words within each sentence, and j # is the index of the correct incremental tree in the candidate list (which is known in the training set) Cost function J can be optimized by gradient descent using backpropagation through structure for computing the gradients [2]. The experiments were run on a portion of the Wall Street Journal Section of the Penn II Treebank. We reserved 100 sentences for training and 500 for testing generalization. On this dataset, the average cardinality of a candidate set F i is 56, while the maximum cardinality is 388. In the ....

C. Goller and A. Kuchler. Learning task-dependent distributed structure-representations by backpropagation through structure. In ICNN96, pages 347--352, 1996.

.... tree is the correct one (but which one is unknown) we must have in this case m i X i=1 y ij = 1: 3) The structured nature of the objects we are dealing with (syntactic trees) suggests adopting a recently introduced class of connectionist architectures, called recursive neural networks (RNNs) [2, 1]. Unlike feedforward neural networks, which are grounded on attribute value representations of data, RNNs can adaptively process labeled graphs, e ectively taking advantage of relations among atomic entities that constitute a single instance. We shall rst give a brief description of the basic RNN ....

....the k th tree. Optimization is solved by gradient descent. In this case, gradients are computed by a special form of backpropagation on the feedforward network obtained by unrolling the state transition network according to the topology of the input tree I . The algorithm was rst proposed in [2] and is referred to as backpropagation through structure (BPTS) The state transition network f( is unrolled to match the topology of the input tree. The output network g( is attached to the replica of f( associated with the root. After recursion (4) is completed, the state vector x(r) at the ....

[Article contains additional citation context not shown here]

C. Goller and A. Kuchler. Learning task-dependent distributed structure-representations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347-352, 1996. 4

....to host the recursive kernel of any non trivial computation [5,6] iii. we manage decision trees as top down counterparts of both mixtures of experts a recent paradigm much used in both neural network [7,8] and computational learning [9] and, in a broad sense, structured data processing [10,11] as will be discussed later in the paper; iv. we manage fuzzy set membership values with a favorite probabilistic interpretation [12] as a computational way of dealing with unsharp shifts between candidate pieces of functions [13] Actually hybrid systems have a long story, dating not ....

....in [27] or by a structured neural network like in [51,69] The scoring network is a multilayer perceptron. A comparison between the learning procedure running on the pair structured neural network, multilayer perceptron a typical instance of BackProtagation Through Structure algorithm [11] and the proposed Back propagation Through Tree may be synthesized as follows: A single neural network is trained to descend a decision tree; in the second procedure the network retains memory of both past queries and network reactions, getting a feed back to its behavior at the end of the ....

C. Goller and A. Kuchler, "Learning task-dependent distributed structurerepresentations by back-propagation through structure", in IEEE International Conference on Neural Networks, pp. 347-352, 1996.

....then plain back propagation can be used on the encoding network. Otherwise, i.e. if there are graphs with cycles, then recurrent back propagation [Pin88] must be used. Consequently, we treat separately these two cases. 4.1. 1 Case I: DAGs This case has been treated by Goller and K uchler in [GK95] Since the training set contains only DAGs, the computation of y W Psi can be realized by backpropagating the error from the feedforward network through the encoding network of each structure. As in back propagation through time, the gradient contributions of corresponding copies of the ....

C. Goller and A. K¨uchler. Learning Task-Dependent Distributed Structure-Representations by Backpropagation Through Structure. AR-report AR-95-02, Institut f¨ur Informatik, Technische Universit ¨at M¨unchen, 1995.

....of DAGs, then plain back propagation can be used on the encoding network. Otherwise, i.e. if there are graphs with cycles, then recurrent back propagation [20] must be used. Consequently, we treat these two cases separately. 5.1. 1 Case I: DAGs This case has been treated by Goller and Kuchler in [12]. Since the training set contains only DAGs, y W Psi can be computed by backpropagating the error from the feed forward network through the encoding network of each structure. As in back propagation through time, the gradient contributions of corresponding copies of the same weight are ....

C. Goller and A. Kuchler. Learning Task-Dependent Distributed Structure-Representations by Backpropagation Through Structure. AR-report AR-95-02, Institut fur Informatik, Technische Universitat Munchen, 1995.

....data structures, like labeled trees and graphs. For example, it has already been shown how neural networks can represent labeled trees [1] and labeled directed graphs [2] and how learning algorithms for recurrent neural networks can be extended to the processing of labeled directed graphs [3] [4], 5] 6] 7] The ability to represent and classify these type of data structures is fundamental in a number of different applications such as medical and technical diagnoses, molecular biology, automated reasoning, software engineering, geometrical and spatial reasoning, and pattern ....

....are not exhaustive since, in general, they can be implemented by any feed forward neural network. In particular, in this paper we will use general feed forward neural networks with across levels connections. For more details on recursive neural networks and associated learning algorithms see [4], 5] C. Encoding of FRA by Boolean Functions Following the idea by Horne Hush [14] a FRAO can be realized by Boolean functions. Let A t = U ; O;X ; ffi; x f ) be a given FRAO, m = jX j be the number of states, l = max(jU j; jOj) be the alphabet size and N be the maximum rank found in U . ....

C. Goller and A. Kuchler, "Learning task-dependent distributed structure-representations by backpropagation through structure", in IEEE International Conference on Neural Networks, 1996, pp. 347--352.

....i.e. which depend on the actual clause occurrence at run time, like the current total number of uses of a given clause in the current search or the number of variables of calling clause becoming instantiated 9 . The results obtained with this network and other standard neural networks models [47] improved the search time of one order of magnitude, however, as is clear from the discussion in Section 5.1 and the examples of features given above, the encoding of structural information in a fixed size vector is not able to capture all the relevant information gathered by symbolic structures ....

....A preliminary move towards this direction is reported in [48] where LRAAM based networks were successfully used to perform classification of symbolic recursive structures encoding logical terms. A refinement of this work led to the definition of the backpropagation through structure algorithm [47]. The experimental comparison between the LRAAM based approach and the backpropagation through structure algorithm reported in [47] shows that the latter algorithm obtains slightly better results for all examples solvable by the LRAAM based approach, but with smaller networks and training times. ....

[Article contains additional citation context not shown here]

C. Goller and A. Kuchler, "Learning task-dependent distributed structure-representations by backpropagation through structure," in IEEE International Conference on Neural Networks, pp. 347--352, 1996.

.... [ Goller et al. 1995b; Goller et al. 1995a ] we developed and implemented a completely new network type that we call folding architecture together with 3 EVALUATION OF PIT, A TOOL FOR PROBABILISTIC REASONING ON INTERVALS 3 the supervised learning method backpropagation through structure [ Goller and Kuchler, 1996 ] This network type is able to process symbolic structures of arbitrary size and it finds the most significant features with respect to the learning task automatically. An experimental comparison with the LRAAM based approach on a set of term classification problems is included with this ....

....with respect to the learning task automatically. An experimental comparison with the LRAAM based approach on a set of term classification problems is included with this evaluation report ( Goller and Kuchler, 1995 ] a shorter version of this paper will appear in the proceedings of the ICNN as [ Goller and Kuchler, 1996 ] We got equally good or slightly better results for all examples solvable by the LRAAM based approach, but the results have been achieved with one order of magnitude smaller networks and training time. Furthermore many examples that could not be solved by LRAAM based networks have been solved ....

C. Goller and A. Kuchler. Learning Task-Dependent Distributed StructureRepresentations by Backpropagation Through Structure. In Proceedings of the ICNN-96, 1996. To appear.

....of arbitrary size cannot be described unambiguously by a finite set of features and the significance of features is extremely task specific. Therefore finding the most important features has to be regarded as part of the learning task. Based on our good results with LRAAM based networks (see [ Goller et al. 1995b; Goller et al. 1995a ] we developed and implemented a completely new network type that we call folding architecture together with 3 EVALUATION OF PIT, A TOOL FOR PROBABILISTIC REASONING ON INTERVALS 3 the supervised learning method backpropagation through structure [ Goller and Kuchler, ....

....cannot be described unambiguously by a finite set of features and the significance of features is extremely task specific. Therefore finding the most important features has to be regarded as part of the learning task. Based on our good results with LRAAM based networks (see [ Goller et al. 1995b; Goller et al. 1995a ] we developed and implemented a completely new network type that we call folding architecture together with 3 EVALUATION OF PIT, A TOOL FOR PROBABILISTIC REASONING ON INTERVALS 3 the supervised learning method backpropagation through structure [ Goller and Kuchler, 1996 ] This network ....

[Article contains additional citation context not shown here]

C. Goller and A. Kuchler. Learning Task-Dependent Distributed StructureRepresentations by Backpropagation Through Structure. AR-Report AR-95-02, Institut fur Informatik, Technische Universitat Munchen, 1995. (a shortened version will appear in the Proceedings of the ICNN-96).

No context found.

C. Goller and A. Kuchler. Learning task-dependent distributed structurerepresentations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347-352, 1996.

No context found.

C. Goller and A. Kuchler. Learning task-dependent distributed structurerepresentations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347-352, 1996.

No context found.

C. Goller and A. Kuechler. Learning Task-dependent Distributed Structure-representations by Back-propagation through Structure. In IEEE International Conference on Neural networks, pages 347--352, 1996.

No context found.

Goller, C., & Kuechler, A. (1996). Learning task-dependent distributed structure-representations by back-propagation through structure. Proceedings of the IEEE International Conference on Neural Networks (pp. 347-- 352). Washington, DC: IEEE.

No context found.

C. Goller and A. Kuchler. Learning task-dependent distributed structure-representations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347-352, 1996. 4

No context found.

C. Goller and A. Kuechler, "Learning task-dependent distributed structure-representations by back-propagation through structure," in IEEE International Conference on Neural networks, 1996, pp. 347--352.

No context found.

C. Goller and A. Kuchler. Learning task-dependent distributed structurerepresentations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347-352, 1996.

No context found.

C. Goller and A. Kuchler. Learning task-dependent distributed structurerepresentations by backpropagation through structure. In IEEE International Conference on Neural Networks, pages 347--352, 1996.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC