R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computation, vol. 4, pp. 406--414, 1992.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... known for several decades [40] 52] 56] More recently, recurrent neural network realizations of finite state automata for recognition and learning of finite state (regular) languages have been explored by numerous authors [3] 8] 14] 16] 18] 37] 60] 64] 66] 67] 82] 87] [102]. There has been considerable work on extending the computational capabilities of recurrent neural network models by providing some form of external memory in the form of a tape [103] or a stack [5] 11] 29] 55] 74] 84] 90] 93] 105] To the best of our knowledge, to date, most of ....

....learning theory, natural language processing, and related areas. The interested reader is referred to [34] 44] 54] and [70] for surveys of grammar inference in general and to [3] 5] 11] 16] 18] 17] 29] 37] 39] 55] 60] 61] 64] 66] 82] 84] 87] 93] and [102] [104] for recent results on grammar inference using neural networks. The neural architecture for syntax analysis that is proposed in this paper does not appear to lend itself to use in grammar inference using conventional neuralnetwork learning algorithms. However, its use in efficient parallel ....

R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent neural networks," Neural Comput., vol. 4, no. 3, p. 406, 1992.

.... ; i = 1; 2; A; 11) and the new symbol s t 2 A = f1; Ag is generated with respect to the distribution P P rob (s t = i) P ; i = 1; 2; A: 12) 4 Extracting stochastic machines from RNNs Transforming trained RNNs into finite state automata has been considered by many [8, 13, 15, 23, 41, 7], usually in the context of training recurrent networks as acceptors of regular languages. After training, the activation space of RNN is partitioned (for example using a vector quantization tool) into a finite set of compartments, each representing a state of a finite state acceptor. Typically, ....

R.L. Watrous and G.M. Kuhn. Induction of finite--state languages using second--order recurrent networks. Neural Computation, 4(3):406--414, 1992.

....computed directly from the machines capture generalization abilities of extracted machines and are related to machines long term behavior. 1 Introduction Researchers have been attracted to the problem of finite state machine (FSM) inference with recurrent neural networks (RNNs) for a long time [2, 4, 5, 11, 13, 15]. RNNs were trained on relatively short sequences and then tested on much longer words to asses the quality of induced temporal structure within the trained networks. It was observed that recurrent neurons activations tent to group in clusters reflecting attractive sets inside the network state ....

R.L. Watrous and G.M. Kuhn. Induction of finite--state languages using second--order recurrent networks. Neural Computation, 4(3):406--414, 1992.

....to tackle this problem. This approach to the induction of asynchronous translators is discussed in connection with other approaches. 1 Introduction In recent years, there has been a lot of interest in training discrete time recurrent neural networks (DTRNN) to behave as finite state machines [7, 9, 13, 17, 18, 20]. This behavior has recently been formalized [3] in response to hard criticisms [12] So far, all work has focused on training DTRNN to behave as finite state acceptors deterministic finite automata or, more generally, as very simple translators such as Mealy machines, or, equivalently, as ....

....a a a a a a S S S ( Delta Delta Delta Delta Delta Delta Delta Delta Delta j j j j Delta Figure 1: The architecture of the second order DTRNN. lines. We build around a second order, single layer network [17, 9, 20, 2, 7]. The network will have N K input units: N for the previous state, and K = j Sigmaj 1 units for one hot (unary) input of symbols from alphabet Sigma plus one unit for a special symbol representing the empty string (i.e. no input) There will be N M 2 output units: N for the next state ....

Watrous, R.L., Kuhn, G.M. (1992) "Induction of Finite-State Languages Using Second-Order Recurrent Networks", Neural Computation 4, 406--414.

....in only a small number of symbols. 5. 2 Recurrent Network Prediction In the past few years several recurrent neural network architectures have emerged which have been used for grammatical inference [10, 23, 21] The induction of relatively simple grammars has been 10 addressed often e.g. [59, 60, 21] on learning Tomita grammars [55] It has been shown that a particular class of recurrent networks are Turing equivalent [50] The set of M symbols from the output of the SOM are linearly encoded into a single input for the Elman network (e.g. if M = 3, the single input is either 1, 0, or 1) ....

R.L. Watrous and G.M. Kuhn. Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4(3):406, 1992.

....in only a small number of symbols. 5. 2 Recurrent Network Prediction In the past few years several recurrent neural network architectures have emerged which have been used for grammatical inference [10, 23, 21] The induction of relatively simple grammars has been 10 addressed often e.g. [59, 60, 21] on learning Tomita grammars [55] It has been shown that a particular class of recurrent networks are Turing equivalent [50] The set of M symbols from the output of the SOM are linearly encoded into a single input for the Elman network (e.g. if M = 3, the single input is either 1, 0, or 1) ....

R.L. Watrous and G.M. Kuhn. Induction of finite state languages using second-order recurrent networks. In J.E. Moody, S.J. Hanson, and R.P Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 309--316, San Mateo, CA, 1992. Morgan Kaufmann Publishers.

.... integrate the connectionist techniques into the symbolic processing framework, giving rise to hybrid parsers such as the Massively Parallel Parsing system [11] the PARSEC model [8] the CDP approach [9] the SPEC architecture [10] the SCAN architecture [14] and the recurrent network models [4, 5, 13]. 2 Confluent Preorder Parser (CPP) In this paper, syntactic parsing is accomplished via a holistic transformation (see [1] and [2] in which the connectionist representation encoding the input sentence is directly mapped to the connectionist representation encoding the target parse tree (Figure ....

R. Watrous and G. Kuhn. Induction of finite state languages using second-order recurrent networks. In J. Moody, S. Hanson, and R. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 309--316. Morgan Kaufmann, 1992.

....and some automatic mechanism must be devised to detect the starting point of each sequence. A network for sequence classification operates like a finite state automaton. Indeed, successful experiments in which a recurrent networks is trained to emulate finite automata have been reported [27][28][29] All these experiments deal with small automata with a limited number of states. Obtaining a solution for complex decisions involving a large number of states is quite a tough problem and inferring automata from examples is a challengy research topic. Techniques like teacher forcing cannot ....

R. Watrous and G. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computation, vol. 4, no. 3, pp. 406--414, 1992.

....the above description does not is really not a difference, since bias units can be added to the framework above by simply clamping an input unit and a context unit to 1:0. Pollack was interested in the task of learning to recognize finite state grammars. Giles, et al. 33] and Watrous and Kuhn [113] both used Pollack s architecture, augmenting it by doing full gradient descent in the network s three dimensional weight matrix with respect to the error generated over multiple time steps. In fact, the network used by Giles, et al. is incremental in that it computes the derivatives of the ....

....Problems with Neural Networks 1 (1 0) no odd zero strings after odd one strings no 000 s pairwise, an even sum of 01 s and 10 s number of 11 s number of 0 s = 0 mod 3 0 1 0 1 Table 4. 1: Regular languages from Tomita [109] used by Pollack [76] Giles, et al. 33] and Watrous and Kuhn [113] for teaching higher order recurrent networks to recognize finitestate grammars. Both positive and negative examples were used for training. input unit can be active at a time. His network allows recurrent connections, so the hidden units can propagate their values to each other from one time ....

Raymond L. Watrous and Gary M. Kuhn. Induction of finite-state languages using second-order recurrent networks. In J. E. Moody, S. J. Hanson, and R. P. Lippman, editors, Advances in Neural Information Processing Systems 4, pages 309--316, San Mateo, California, 1992. Morgan Kaufmann Publishers. Bibliography 127

.... state automata have been shown to be useful for modeling fuzzy dynamical systems, often in conjunction with recurrent neural networks [42, 43, 44, 35, 45] There has been much work on the learning, synthesis, and extraction of finite state automata in recurrent neural networks, see for example [46, 47, 48, 49, 50, 51, 52, 53]. A variety of neural network implementations of FFA have been proposed [39, 54, 40, 55] We have previously shown how fuzzy finite state automata can be mapped into recurrent neural networks with second order weights using a crisp representation 2 of FFA states [56] That encoding required a ....

R.L. Watrous and G.M. Kuhn, "Induction of finite-state languages using second-order recurrent networks, " Neural Computation, vol. 4, no. 3, pp. 406, 1992.

....based on a second order recurrent neural network may be trained to behave as an instance of the most powerful class of deterministic sequential translator. 1 Introduction In recent years, there has been a lot of interest in training recurrent neural networks to behave as finite state machines [3, 4, 7, 9]. So far, all work has focused on training recurrent neural networks to behave as finitestate acceptors deterministic finite automata or, more generally, very simple translators: such as Mealy machines, or, equivalently, as Moore machines[5] A Mealy machine is a six tuple M = Q; Sigma; ....

Watrous, R.L. and Kuhn, G.M. (1992) "Induction of Finite-State Languages Using Second-Order Recurrent Networks", Neural Computation 4, 406--414.

....in only a small number of symbols. 5. 2 Recurrent Network Prediction In the past few years several recurrent neural network architectures have emerged which have been used for grammatical inference [10, 23, 21] The induction of relatively simple 13 grammars has been addressed often e.g. [59, 60, 21] on learning Tomita grammars [55] It has been shown that a particular class of recurrent networks are Turing equivalent [50] The set of symbols from the output of the SOM are linearly encoded into a single input for the Elman network (e.g. if r , the single input is either 1, 0, or 1) The ....

R.L. Watrous and G.M. Kuhn. Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4(3):406, 1992.

....in only a small number of symbols. 5. 2 Recurrent Network Prediction In the past few years several recurrent neural network architectures have emerged which have been used for grammatical inference [10, 23, 21] The induction of relatively simple 13 grammars has been addressed often e.g. [59, 60, 21] on learning Tomita grammars [55] It has been shown that a particular class of recurrent networks are Turing equivalent [50] The set of symbols from the output of the SOM are linearly encoded into a single input for the Elman network (e.g. if r , the single input is either 1, 0, or 1) The ....

R.L. Watrous and G.M. Kuhn. Induction of finite state languages using second-order recurrent networks. In J.E. Moody, S.J. Hanson, and R.P Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 309--316, San Mateo, CA, 1992. Morgan Kaufmann Publishers.

....If instead of using the network itself, an automaton is extracted from the network after training and the transition probabilities of the extracted automaton are estimated from the sample, the relative entropy with respect to the true distribution is reduced. 1 Introduction A number of papers [1, 2, 3, 4] have explored the ability of second order recurrent neural networks to infer simple regular grammars from complete (positive and negative) training samples. The generalization performance of a finite state automaton extracted from the trained network is better than that of the network itself [1, ....

Watrous, R.L., Kuhn, G.M.: "Induction of Finite-State Languages Using SecondOrder Recurrent Networks" Neural Computation 4 (1992) 406--414.

....Firenze Via di Santa Marta 3 50139 Firenze Italy Tel. 39 (55) 479.6265 Fax 39 (55) 479.6363 e mail : marco ingfi1.ing.unifi.it Abstract Many researchers have recently explored the use of recurrent networks for the inductive inference of regular grammars from positive and negative examples [5, 9, 11] with very promising results. In this paper, we give a set of weight constraints guaranteeing that a recurrent network behave as an automaton and show that the measure of this admissible set decreases progressively as the network dimension increases, thus suggesting that automata behavior becomes ....

.... suggest looking for more valuable approaches based on the divide et impera paradigm that allow us to limit the network dimensions [3] 1 Introduction Recently, many researchers have used recurrent neural networks for performing inductive inference of regular grammars with very promising results [5, 9, 11]. Unlike commonly approached neural network tasks, in problems of inductive inference, one is not only interested in exhibiting a satisfactory generalization to new examples, but also in capturing the rule hidden in the examples. This requires approaching a very hard task, that is the ....

[Article contains additional citation context not shown here]

R.L. Watrous and G.M. Kuhn, "Induction of Finite-State Languages Using Second-Order Recurrent Networks," Neural Computation Vol. 4, No. 3, May 1992, pp.406-414. 10

....a recurrent network can learn to correctly classify all strings of a training set, the internal DFA representation can become unstable leading to misclassification of long strings that were not part of the training set. Similar observations have also been reported elsewhere in the literature [17, 29, 31]. Various methods for stabilizing the internal DFA representation during training have been proposed. These range from discrete space recurrent networks to the introduction of error functions which favor binary internal representations [8, 9, 13, 31] However, it is not at all clear whether these ....

....extension of [6] in that complete deterministic finite state automata are extracted from recurrent networks. 3 Dynamic Space Exploration 3. 1 Observation We describe our heuristic for extracting rules from recurrent networks in the form of DFAs [11] Different approaches are described in [6, 9, 29, 31]. A preliminary method for extracting rules when the source grammar is unknown has been discussed in [12] We will focus on the following question: How good are the rules extracted from recurrent networks The algorithm used to extract a finite state automaton from a network is based on the ....

R. Watrous and G. Kuhn, "Induction of finite state languages using secondorder recurrent networks," in Advances in Neural Information Processing Systems 4 (J. Moody, S. Hanson, and R. Lippmann, eds.), (San Mateo, CA), pp. 309--316, Morgan Kaufmann Publishers, 1992.

....formal languages from examples. The resulting networks often displayed complex limit dynamics whichwere fractal in nature #Kolen, 1993#. Alternative architectures had been employed earlier for related tasks #Jordan, 1986, Pollack, 1987, Cleeremans et al. 1989#. Others have been proposed since #Watrous Kuhn, 1992, Frasconi et al. 1992, Zeng et al. 1994, Das Mozer, 1994, Forcada Carrasco, 1995# and a number of approaches to analysing recurrent networks have been developed. One of the principal themes has been the use of clustering and minimization techniques to extract a Finite State Automaton #FSA# ....

Watrous, R.L. & G.M. Kuhn, 1992. Induction of Finite State Languages Using Second-Order Recurrent Networks, Neural Computation 4#3#, 406#414.

....L#A # in the following manner: for any string #=# 1 : # T 2A # , let x T = f #T # : #f # 1 #x 0 #. Then # is accepted if x T 2 X yes , rejected if x T 2 X no . This framework, introduced in #Pollack, 1991# has been extended in a number of directions both experimental #Giles et al. 1992, Watrous Kuhn, 1992# and theoretical #Moore, 1997, Casey, 1996#. A variety of methods have been developed for extracting a deterministic #nite automaton #DFA# from the dynamical recognizer once it is trained #Omlin Giles, 1996, Das Mozer, 1994, Manolios Fanelli, 1994, Zeng et al. 1994# although the recognizer ....

....# 15, and classi#ed by a randomly generated DFA with 65 states. We believe this kind of data would present a serious challenge for earlier architectures, which had generally relied on datasets in which shorter strings were overrepresented #Tomita, 1982# classi#ed byDFA s with many fewer states #Watrous Kuhn, 1992# or with a special kind of internal structure #Clouse et al. 1997#. The training was done in batch mode, using a fully parallel algorithm on a MasPar MP 2, with two processors devoted to each string. Experiment 1: The Orthogonal Case In our #rst experiment, we set # = 0 which implies that A # ....

Watrous, R.L. & G.M. Kuhn, 1992. Induction of Finite State Languages Using Second-Order Recurrent Networks, Neural Computation 4#3#, 406#414.

....i Inform atica Industrial, UPC CSIC, Barcelona alquezar lsi.upc.es, asanfeliu iri.upc.es, msainz iri.upc.es Abstract In this paper, the ability of recurrent neural networks (RNNs) for regular inference (RI) from positive and negative examples is investigated. As in some previous works [1, 2], RNNs were trained to learn the string classification task from samples of some target regular languages. In addition, an automaton extraction method [3] was applied to each one of the trained nets to obtain a description of the inference outcome. For comparison purposes, a symbolic RI method, ....

....of learning a regular language, or a finite state automaton (FSA) accepting it, from string examples or queries. For the case of RI from positive and negative samples, a theoretical framework has been presented in [5] and several methods have been proposed, using symbolic [4, 6] connectionist [1, 2, 3, 7] and genetic [8] approaches. However, the comparative performance of these methods on benchmark tests has not been assessed sufficiently. In this work, both first and second order Augmented Single Layer Recurrent Neural Networks (ASLRNNs) 1 [3, 7] have been trained to learn the string ....

[Article contains additional citation context not shown here]

R.L. Watrous and G.M. Kuhn, "Induction of finite state languages using second-order recurrent networks," Neural Computation 4, pp.406-414, 1992.

....with A2 solves it within 2797 trials. We also ran another experiment with architecture A2, but without self connections for hidden units. Guessing solved the problem within 250 trials on average. Tomita grammars. Many authors also use Tomita s grammars [30] to test their algorithms. See, e.g. [2, 32, 22, 17, 16]. Since we already tested parity problems above, we now focus on a few parity free Tomita grammars (nr.s #1, #2, #4) Previous work even facilitated the problems by restricting sequence length. e.g. in [17] maximal test (training) sequence length is 15 (10) Reference [17] reports the number ....

R. L. Watrous and G. M. Kuhn. Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4:406--414, 1992.

....for grammar induction like deterministic finite state automata, boolean formula or propositional logic. More recently, Pollack [79] proposed dynamical recognizers as an interesting alternative to those symbolic approaches, leading to a wide range of Recurrent Neural Network (RNN) architectures [109, 115, 23, 29] that had been employed for similar tasks. However, none of them could compete in the Abbadingo competition because of the proposed problems size. The application of the SAGE search algorithm to this problem has been motivated by the Abbadingo competition organized by Lang and Pearlmutter [54] It ....

R. L. Watrous and G. M. Kuhn. Induction of finite state languages using secondorder recurrent networks. Neural Computation, 4(3):406--414, 1992.

....automata have been the main paradigm of temporal symbolic knowl 1 2 3 4 5 6 7 8 9 10 Figure 1: Example of DFA: Shown is a minimal, randomly generated DFA with 10 states and 2 input symbols. State 1 is the DFA s start state; accepting states are shown with double circles. edge extraction [4, 8, 14, 20, 22]. 2 Finite State Automata A large class of discrete processes can be modeled by deterministic finite state automata (DFAs) They form the basic building blocks of theoretical models of computation. More powerful models of computation can be obtained by adding new elements to DFAs; restrictions on ....

R. Watrous and G. Kuhn, "Induction of finitestate languages using second-order recurrent networks, " Neural Computation, vol. 4, no. 3, p. 406, 1992.

....is simply a single layer of nodes. Even models which appear very different (Back and Tsoi, 1991; Elman, 1990; Jordan and Rumelhart, 1992; Narendra and Parthasarathy, 1990) can be cast into this framework. Our results will not be applicable to networks with high order terms (Giles et al. 1990; Watrous and Kuhn, 1992). All of the bounds described in this paper will be derived for multilayer architectures. However, an L layer network can always be implemented as a single layer, fully connected model, such as the Hopfield model, if one is willing to allow an L time step slowdown for each single time step of ....

Watrous, R. and Kuhn, G. (1992). Induction of finite--state languages using second--order recurrent networks. Neural Computation, 4(3):406--414.

....in IEEE Trans. on Neural Networks vol. 5, no. 5, p. 848, 1994. Copyright IEEE. 2 PRUNING A RECURRENT NETWORK To test our pruning heuristic, we incrementally trained discrete time, fully recurrent temporally driven neural networks with second order weights W ijk to learn regular languages [1, 4, 13, 15]. The network accepts a time ordered sequence of inputs and evolves with dynamics defined by the following equations: S (t 1) i = g( Xi i b i ) Xi i j P j;k W ijk S (t) j I (t) k ; where g is a sigmoid discriminant function and b i is the bias associated with hidden recurrent state ....

.... recurrent learning (RTRL) algorithm for recurrent neural networks ( 16] For more details see [4] The heuristic used for extracting rules from recurrent networks in the form of deterministic finite state automata (DFA s) is described in detail in [4] Different approaches are discussed in [1, 15]. The quality of the extracted rules has been discussed in [5] The algorithm used to extract a finite state automaton from a network is based on the observation that the outputs of the recurrent state neurons of a trained network tend to cluster. Our hypothesis is that collections of these ....

R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks, " Neural Computation, vol. 4, no. 3, p. 406, 1992.

....one accepting state, the network state layer is not suitable for a direct output computation, unless one neuron of the layer is reserved for that task. In order to reach the state coded by this neuron, a possible solution is that of adding one more end symbol to each string (Giles et al. 1992b; Watrous Kuhn, 1992). 4.2. The automata weight space In the previous section, we have discussed of automata realization by considering boolean values. A straightforward consequence of this theoretical assumption is that the weights connecting the radial basis functions to the sigmoidal neurons must be infinite. ....

....automata behavior arising from the learning of sequences of relatively small length, may change to more complex dynamics for longer sequences. To some extent, the techniques for extracting automata after learning (Cleeremans et al. 1989; Servan Schreiber et al. 1991; Giles et al. 1992b; Watrous and Kuhn, 1992) are interesting attempts to overcome this problem. For example, Giles et al. 1992b) report explicitly that the extracted automaton can exhibit better performance than the recurrent network from which it has been extracted. However, an implicit assumption for a successful extraction of automata ....

[Article contains additional citation context not shown here]

Watrous, R. L. and Kuhn, G. M. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4(3):406--414.

....computed directly from the machines capture generalization abilities of extracted machines and are related to machines long term behavior. 1 Introduction Researchers have been attracted to the problem of finite state machine (FSM) inference with recurrent neural networks (RNNs) for a long time [2, 4, 5, 11, 13, 15]. RNNs were trained on relatively short sequences and then tested on much longer words to asses the quality of induced temporal structure within the trained networks. It was observed that recurrent neurons activations tent to group in clusters reflecting attractive sets inside the network state ....

R.L. Watrous and G.M. Kuhn. Induction of finite--state languages using second--order recurrent networks. Neural Computation, 4(3):406--414, 1992.

....x UCSD CSE Dept. gcottrell ucsd.edu 1 Introduction In the neural network language induction literature, induction of finite state automata is commonly thought of as the domain of recurrent network architectures [Cleeremans et al. 1989] Elman, 1991] Pollack, 1991] Giles et al. 1992] [Watrous and Kuhn, 1992]. However, recent work [Giles et al. 1994] has shown that a restricted class of recurrent nets can learn a subclass of finite state automata called finite memory automata. In this paper, we show that feedforward only architectures can represent and learn a class of automata, DeBruijn automata. ....

Watrous, R. and Kuhn, G. (1992). Induction of finite state languages using second-order recurrent networks. In [Moody et al., 1992], pages 309-- 316.

....existence of extreme DFAs for which the weight strength scales with DFA size. 1 INTRODUCTION It is possible to train recurrent neural networks to behave like deterministic finite state automata [Elman, 1990, Frasconi et al. 1991, Giles et al. 1992, Pollack, 1991, Servan Schreiber et al. 1991, Watrous and Kuhn, 1992] The internal representation of learned DFA states can deteriorate due to the dynamical nature of recurrent networks making predictions about the generalization performance of trained recurrent networks difficult [Zeng et al. 1993] Methods for constructing DFAs in recurrent networks with ....

Watrous, R. and Kuhn, G. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4(3):406.

....problem. We also made the problem very learnable in the sense that we gave the neural network ample strings for effective training. For network architecture we chose a second order discrete time recurrent network since such networks have demonstrated good performance on such problems [6, 14, 19]. As such our experiments consist of 500 simulations for each data point and achieve useful (90 ) confidence levels. 3 NOISE INJECTION IN RECURRENTNEURALNETWORKS 3.1 Noiseless Network Architecture and Training We briefly review the second order fully recurrent neural network, string (sequence) ....

R.L. Watrous and G.M. Kuhn. Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4(3):406, 1992.

....formal languages from examples. The resulting networks often displayed complex limit dynamics which were fractal in nature (Kolen, 1993) Alternative architectures had been employed earlier for related tasks (Jordan, 1986, Pollack, 1987, Cleeremans et al. 1989) Others have been proposed since (Watrous Kuhn, 1992, Frasconi et al. 1992, Zeng et al. 1994, Das Mozer, 1994, Forcada Carrasco, 1995) and a number of approaches to analysing recurrent networks have been developed. One of the principal themes has been the use of clustering and minimization techniques to extract a Finite State Automaton (FSA) ....

Watrous, R.L. & G.M. Kuhn, 1992. Induction of Finite State Languages Using Second-Order Recurrent Networks, Neural Computation 4(3), 406--414.

....networks (LRNNs) here, are primarily feedforward networks, with the exception that feedback connections exist between some limited sets or layers of units. Up to now, almost all RNN models proposed for formal language learning are LRNNs, including both first order [3, 1, 4, 7] and second order [5, 2, 8, 12] networks. LRNNs are easier to analyze because of their limited feedback connections and layered structures. These networks have been demonstrated to be capable of solving some nontrivial formal language learning problems. In this paper, our focus is on using first order LRNNs for inductive ....

....temporal sequence processing problems are of this nature. Preliminary results based on the ASCOC model give 85 90 accuracy in detecting and classifying the subpatterns correctly. While the focus of this study is on first order LRNNs, we are also aware of some second order LRNN models (e.g. [5, 2, 8, 12]) that have also been applied to grammatical inference problems. It is our intention to perform extensive comparison of different RNN models in our future work. Acknowledgement The research work reported in this paper was made possible by research grants from the Sino Software Research Centre ....

R.L. Watrous and G.M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computation, vol. 4, pp. 406--414, 1992.

....being trained on exemplars drawn from the grammar. ii) It was able to remember length constraints on the strings. iii) Long distance contingencies could be remembered given certain constraints. Other notable efforts on grammatical inference using second order recurrent networks are presented in [5, 14]. This paper attempts to analyze the internal representations that develop in an experiment similar to the one above, in an effort to explain the prediction capabilities of the network. It also tries to show the relationship that exists between the internal representations and the amount of ....

R.L. Watrous and G.M. Kuhn. Induction of finite-state languages using second-order recurrent networks. In S.J. Hanson J.E. Moody and R.P. Lippmann, editors, Advances in Neural Information Processing Systems -4. Morgan Kaufmann, San Mateo, CA, 1992.

.... Connectionist representation of the target parse tree Figure 3: Holistic parsing paradigm Compared with other grammatical inference approaches, CPP is unique in several ways : ffl Syntactic parsing as addressed by CPP is significantly more difficult than the classification problem tackled by [8, 23, 31], and the prediction problem attempted by [7, 18, 26] Firstly, CPP aims at tackling context free grammars instead of regular languages. In general, to parse context free languages, external memory is required [13] In contrast to other approaches (such as SPEC [19] and the Neural Network ....

R. Watrous and G. Kuhn. Induction of finite state languages using second-order recurrent networks. In J. Moody, S. Hanson, and R. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 309--316, San Mateo, CA, 1992. Morgan Kaufmann.

....are used. Continuous sigmoids offer other advantages besides their use in gradientbased training algorithms; they also permit analog VLSI implementation, the 1 See, for example, Elman [1990] Frasconi et al. 1991] Giles et al. 1991; 1992] Pollack [1991] Servan Schreiber et al. 1991] and Watrous and Kuhn [1992]. 938 C. W. OMLIN AND C. L. GILES foundations necessary for the universal approximation theories of neural networks, the interpretation of neural network outputs as a posteriori probability estimates, etc. For more details, see Haykin [1994] Stability of an internal DFA state representation ....

WATROUS, R., AND KUHN, G. 1992. Induction of finite-state languages using second-order recurrent networks. Neural Comput. 4, 3, 406.

....map) is used to extract meaningful information from another network (recurrent neural network) 1 Introduction Considerable interest has been shown in language inference using neural networks. Recurrent networks were shown to be able to learn small regular languages (Das and Das, 1991; Watrous and Kuhn, 1992; Giles et al. 1992; Zeng et al. 1994) The recurrent nature of these networks is able to capture the dynamics of the underlying computation automaton (Das, Giles and Sun, 1992) Hidden units activations represent past histories and clusters of these activations can represent the states of the ....

.... The training of the first order recurrent neural networks that recognize finite state languages is discussed in (Elman, 1990) where the results were obtained by training the network to predict the next symbol, rather than by training the network to accept or reject strings of different lengths (Watrous and Kuhn, 1992). The problem of inducing languages from examples has also been approached using second order recurrent networks (Giles et al. 1992; Watrous and Kuhn, 1992) The orientation of this work is somewhat different. An initial mealy machine (IMM) transforms a nonempty word over an input alphabet to a ....

[Article contains additional citation context not shown here]

Watrous, R. L., and Kuhn, G. M. 1992. Induction of Finite-State Languages Using Second-Order Recurrent Networks. Neural Comp. 4, 406-414.

.... Bengio (1995) Lin et al. 1995) For the same purpose, some papers also use the socalled parity problem , e.g. Bengio et al. 1994) Bengio and Frasconi (1994) Some of Tomita s grammars (1982) are also often used as benchmark problems for recurrent nets (see, e.g. Bengio and Frasconi, 1995; Watrous and Kuhn, 1992; Pollack, 1991; Miller and Giles, 1993; Manolios and Fanelli, 1994) This paper exemplifies: such problems can be solved more quickly by random weight guessing than by the proposed algorithms. Guessing. With a given architecture, random weight guessing works as follows: REPEAT randomly ....

....We also ran another experiment with architecture A2, but without self connections for the hidden units. Guessing solved the problem within 250 trials on average. Tomita grammars. Many authors also use Tomita s grammars (1982) to test their algorithms. See, e.g. Bengio and Frasconi (1995) Watrous and Kuhn (1992), Pollack (1991) Miller and Giles (1993) Manolios and Fanelli (1994) Since we already tested parity problems above, we now focus on a few parity free Tomita grammars (nr.s #1, #2, #4) Previous work facilitated the problems by restricting sequence length. e.g. Miller and Giles maximal test ....

Watrous, R. L. and Kuhn, G. M. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4:406--414.

....weighted sum of neuron inputs. Then, the structure is transformed into the space of neuron activations (section 4) There are many applications where oscillatory dynamics of recurrent networks is desirable. For example, when trained to act as a finite state machine ( 7] 8] 9] 10] 14] 16] [17], 18] the network has to induce a stable representation of state transitions associated with each input symbol of the machine. A transition may have a character of a loop (do not move from the current state when the symbol x is presented) or a cycle (when repeatedly presenting the same input, ....

R.L. Watrous and G.M. Kuhn. Induction of finite--state languages using second--order recurrent networks. Neural Computation, 4(3):406--414, 1992.

....high order back propagation network, but had difficulties with convergence on Tomita s languages TL2 and TL6. Other work has applied neural networks to the induction of higher order Chomsky languages (e.g. context free and Turing) with varying success ( Giles et al. 1990] Giles et al. 1992] [Waltrous and Kuhn 1992]; Williams and Zisper 1988] The majority of these methods, however, require some part of the automata structure to be specified by the user before being applied. In particular, the user almost always has to specify the number of states. Genetic programming (GP) Koza 1992] provides the ....

Waltrous, R. L. and Kuhn, G. M. (1992). Induction of finitestate languages using second-order recurrent networks, Neural Computation. Volume 4. Pages 404-414.

....between lower level units. Instead of combining together behaviors as was done with Method I, these units represent the degree to which one behavior should follow another at any particular time they therefore resemble higher order units to some extent (c.f. Pollack, 1991; Giles, 1992; and Watrous, 1992). Take for example Figure 2a. In one case (position 9) the animat should go south when it senses light, while in another case (position 5) it should go north. To decide whether to move north or south after the light, it is sufficient to know whether the animat sensed heat or cold in the previous ....

Watrous, R. L. and Kuhn, G. M. (1992). Induction of finitestate languages using second-order recurrent networks.

....for grammar induction like deterministic finite state automata, boolean formula or propositional logic. More recently, Pollack [16] proposed dynamical recognizers as an interesting alternative to those symbolic approaches, leading to a wide range of Recurrent Neural Network (RNN) architectures [18, 19, 4, 6] that have been employed for similar tasks. However, none of them could compete in the Abbadingo competition because of the proposed problems size. 3.2 The Abbadingo Competition The Abbadingo competition (organized by Lang and Pearlmutter [14] is a challenge proposed to the machine learning ....

R. L. Watrous and G. M. Kuhn. Induction of finite state languages using secondorder recurrent networks. Neural Computation, 4(3):406--414, 1992.

....of labeled strings and is requested to infer a set of rules that define a formal language. It can be considered as a prototype for more complex language processing problems. However, even in the simplest case, i.e. regular grammars, the task can be proved to be NPcomplete [2] Many researchers [3, 4, 5] have approached grammatical inference with recurrent networks. These studies demonstrate that second order networks can be trained to approximate the behavior of finite state automata (FSA) However, memories learned in this way appear to lack robustness and noisy dynamics become dominant for ....

....second order networks can be trained to approximate the behavior of finite state automata (FSA) However, memories learned in this way appear to lack robustness and noisy dynamics become dominant for long input strings. This has motivated research to extract automata rules from the trained network [3, 5]. In many cases, it has been shown that the extracted automaton outperforms the trained network. Although FSA extraction procedures are relatively easy to devise for symbolic inputs, they may be more difficult to apply in tasks involving a subsymbolic or continuous input space, such as in speech ....

[Article contains additional citation context not shown here]

R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computation, vol. 4, no. 3, pp. 406--414, 1992.

No context found.

R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computation, vol. 4, pp. 406--414, 1992.

No context found.

Watrous, R.L. & G.M. Kuhn, 1992. Induction of Finite State Languages Using Second-Order Recurrent Networks, Neural Computation 4(3), 406--414.

No context found.

R. L. Watrous and G. M. Kuhn. Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4:406--414, 1992.

No context found.

R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks", Neural Computation, 4:406--414 (1992).

No context found.

Watrous, R.L.; Kuhn, G.M. "Induction of Finite-State Languages Using Second-Order Recurrent Networks" Neural Computation 4, 469-490.

No context found.

R. L. Watrous and G. M. Kuhn. Induction of finite-state languages using secondorder recurrent networks. Neural Computation, 4(3):406--414, 1992.

No context found.

Raymond L. Watrous and Gary M. Kuhn. Induction of finite-state languages using second-order recurrent networks. In J. E. Moody, S. J. Hanson, and R. P. Lippman, editors, Advances in Neural Information Processing Systems 4, pages 309--316, San Mateo, California, 1992. Morgan Kaufmann Publishers. Bibliography 127

No context found.

R.L. Watrous & G.M. Kuhn. (1992) Induction of finite state languages using second-order recurrent networks. In J.E. Moody, S.J. Hanson, & R.P. Lippmann (eds.), Advances in Neural Information Processing Systems 4, 969-976. San Mateo, CA: Morgan Kaufmann.

No context found.

R.L. Watrous, G.M. Kuhn, Induction of Finite-State Languages Using Second-Order Recurrent Networks, Neural Computation, accepted for publication (1992a) and these proceedings, (1992b).

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC