P. Manolios and R. Fanelly, "First order recurrent neural networks and deterministic finite state automata," Neural Computation, vol. 6, pp. 1155--1173, 1994.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....information latching is of special signi cance for the recurrent network community applying their models to grammar automata inference tasks. A technique widely used in this community to stabilize the network performance is to extract nite state representations from trained networks [7] 8] 9] [10] [11] 12] 13] 14] Usually, clusters in recurrent neurons activation space are identi ed with states of the extracted machines [15] This methodology was analyzed by Casey [13] and Ti no et al. 16] Ti no and K otele s [17] showed that extracting nite state representations from RNNs trained ....

P. Manolios and R. Fanelli, \First order recurrent neural networks and deterministic nite state automata," Neural Computation, vol. 6, no. 6, pp. 1155-1173, 1994.

.... ; i = 1; 2; A; 11) and the new symbol s t 2 A = f1; Ag is generated with respect to the distribution P P rob (s t = i) P ; i = 1; 2; A: 12) 4 Extracting stochastic machines from RNNs Transforming trained RNNs into finite state automata has been considered by many [8, 13, 15, 23, 41, 7], usually in the context of training recurrent networks as acceptors of regular languages. After training, the activation space of RNN is partitioned (for example using a vector quantization tool) into a finite set of compartments, each representing a state of a finite state acceptor. Typically, ....

P. Manolios and R. Fanelli. First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6(6):1155--1173, 1994.

....3. 3 Extracting finite state machines from trained networks Once successfully trained, specialized algorithms may extract FSM from the dynamics of the DTRNN; some use a straightforward equipartition of neural state space followed by a branch and bound algorithm [12] or a clustering algorithm [10, 16, 20]. Very often, the finite state automaton extracted behaves correctly for strings of any length, even better than the original DTRNN. But automaton extraction algorithms have been criticised [24, 25] in the sense that FSM extraction may not reflect the actual computation performed by the DTRNN. ....

P. Manolios and R. Fanelli. First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6(6):1154--1172, 1994.

....are treated as exceptions. The results show that, in this way, the DTRNN usually manages to learn a simplified approximate language. 1 Introduction Discrete time recurrent neural networks (DTRNN) may be trained to to recognize regular languages from sets of example and counterexample strings [3, 11, 5, 13, 7, 4], under the intuitive assumption that, being state machines themselves, Work supported through grants TIC97 0941 and HI1996 0055 of the Spanish government. DTRNN can emulate deterministic finite automata (DFA) that is, regular language recognizers. DTRNN can be indeed trained to behave as ....

P. Manolios, R. Fanelli. First order recurrent neural networks and deterministic finite state automata. Neural Computation 6(6):1154--1172, 1994.

....to tackle this problem. This approach to the induction of asynchronous translators is discussed in connection with other approaches. 1 Introduction In recent years, there has been a lot of interest in training discrete time recurrent neural networks (DTRNN) to behave as finite state machines [7, 9, 13, 17, 18, 20]. This behavior has recently been formalized [3] in response to hard criticisms [12] So far, all work has focused on training DTRNN to behave as finite state acceptors deterministic finite automata or, more generally, as very simple translators such as Mealy machines, or, equivalently, as ....

Manolios, P., Fanelli, R. (1994) "First-Order Recurrent Neural Networks and Deterministic Finite State Automata", Neural Computation, 6, 1155--1173.

....of di erential equations is computed iteratively by use of a time constant (usually 1) This will lead to an insu cient accuracy, especially if real time synchronisation is required as presented in the application example in Section 3. Only systems with discrete states (e.g. nite state automata [20]) can be computed appropriately. To improve the quality of the propagation process, a smaller time constant dt or modi ed propagation methods can be used. A discussion of numerical methods for the computation of (recurrent) neural networks can be found, for example, in [21] A propagation method ....

P. Manolios, R. Fanelli, First-order recurrent neural networks and deterministic "nite state automata, Neural Comput. 6 (6) (1994) 1155}1173.

.... Moore machines and connectionist preference Moore machines already support very important general properties of language and they form the basis in the Chomsky hierarchy [Hopcroft and Ullman, 1979] In contrast to other research work in the area of nite automata and connectionist networks [Manolios and Fanelli, 1994, Omlin and Giles, 1994, Omlin and Giles, 1996] we do not only want to learn an acceptor which learns to accept a correct input sequence but we are interested in building robust learning preference Moore machines which can produce output. Furthermore, an important aspect of our work is that we ....

Manolios, P. and Fanelli, R. (1994). First-order recurrent neural networks and deterministic nite state automata. Neural Computation, 6:1155-1173.

....within 250 trials on average. Again it should be mentioned that different input representations and noise types may lead to worse RS performance (Yoshua Bengio, personal communication, 1996) Tomita grammars. Many authors also use Tomita s grammars [18] to test their algorithms. See, e.g. [2, 19, 14, 11, 10]. Since we already tested parity problems above, we now focus on a few parity free Tomita grammars (nr.s #1, #2, #4) Previous work facilitated the problems by restricting sequence length. e.g. in [11] maximal test (training) sequence length is 15 (10) Reference [11] reports the number of ....

P. Manolios and R. Fanelli. First-order recurrent neural networks and deterministic finite state automata. Neural Computation, 6:1155--1173, 1994.

....in an analog state space and hence, the analog neuromaton implementation can serve as an initial network con guration for the underlying learning procedure. The resulting transition rules may be extracted after the neuromaton is adapted to training data. This approach has widely been applied [4, 5, 7, 16, 19, 20, 27, 28], especially for second order neural nets which are also related to recurrent RBF networks [6] The analog implementations of neuromata o er other advantages besides their use in gradient based training algorithms; they also permit analog VLSI implementation, the foundations necessary for the ....

Manolios, P., Fanelli, R. First-order recurrent neural networks and deterministic nite state automata. Neural Computation, 6, 1155-1173, 1994.

....more advanced computational tasks. Under this intuitive assumption, a number of researchers set out to test whether sigmoid DTRNN could learn FSM behavior from samples (Cleeremans et al. 1989; Pollack 1991; Giles et al. 1992; Watrous Kuhn 1992; Maskara Noetzel 1992; Sanfeliu Alqu ezar 1994; Manolios Fanelli 1994; Forcada Carrasco 1995; Ti no Sajda 1995; Neco Forcada 1996; Gori et al. 1998) The results show that, indeed, DTRNN can learn FSM like behavior from samples, but some problems persist: after learning, FSM like behaviour is observed only for for short input sequences but degrades with ....

.... the processing (Bengio et al. 1994) Once trained, specialized algorithms may extract FSM from the dynamics of the DTRNN; some use a straightforward equipartition of neural state space followed by a branch and bound algorithm (Giles et al. 1992) or a clustering algorithm (Cleeremans et al. 1989; Manolios Fanelli 1994; Gori et al. 1998) Very often, the nite state automaton extracted behaves correctly for strings of any length. Automaton extraction algorithms have been criticised (Kolen Pollack 1995; Kolen 1994) in the sense that FSM extraction may not re ect the actual computation performed by the DTRNN. ....

Manolios, P., & R. Fanelli. 1994. First order recurrent neural networks and deterministic nite state automata. Neural Computation 6(6):1154-1172.

....in an analog state space and hence, the analog neuromaton implementation can serve as an initial network con guration for the underlying learning procedure. The resulting transition rules may be extracted after the neuromaton is adapted to training data. This approach has widely been applied [4, 5, 7, 16, 19, 20, 27, 28], especially for second order recurrent neural networks which are also related to recurrent RBF networks [6] The analog implementations of neuromata o er other advantages besides their use in gradient base training algorithms; they also permit an analog VLSI implementation, foundations necessary ....

Manolios, P., Fanelli, R. First-order recurrent neural networks and deterministic nite state automata. Neural Computation, 6, 1155-1173, 1994.

....both experimental #Giles et al. 1992, Watrous Kuhn, 1992# and theoretical #Moore, 1997, Casey, 1996#. A variety of methods have been developed for extracting a deterministic #nite automaton #DFA# from the dynamical recognizer once it is trained #Omlin Giles, 1996, Das Mozer, 1994, Manolios Fanelli, 1994, Zeng et al. 1994# although the recognizer will sometimes induce a non regular language which cannot be modeled exactly byanyDFA #Blair Pollack, 1997, Kolen, 1993#. Many of these approaches have used a neural network framework in which the state space X is a hypercube and each map f # is a ....

Manolios, P.&R.Fanelli, 1994. First order recurrent neural networks and deterministic #nite state automata, Neural Computation 6#6#, 1155#1173.

.... machines (FSM) a number of researchers set out to test whether DTRNN with real valued, continuous sigmoid activation functions could learn FSM behavior from samples (Cleeremans et al. 1989; Pollack 1991; Giles et al. 1992; Watrous Kuhn 1992; Maskara Noetzel 1992; Sanfeliu Alqu ezar 1994; Manolios Fanelli 1994; Forcada Carrasco 1995; Ti no Sajda 1995; Neco Forcada 1996; Gori et al. 1998) and even compared the performance of di erent architectures (Miller Giles 1993; Horne Giles 1995) The use of sigmoid functions is motivated by the need to have error functions which are di erentiable ....

.... Once the DTRNN has been trained, researchers use specialized algorithms to extract FSM from the dynamics of the DTRNN; some use a straightforward equipartition of neural state space followed by a branch and bound algorithm (Giles et al. 1992) or a clustering algorithm (Cleeremans et al. 1989; Manolios Fanelli 1994; Gori et al. 1998) Very often, the nite state automaton extracted behaves correctly, and then does so, obviously, for strings of any length. However, automaton extraction algorithms have been criticised (Kolen Pollack 1995; Kolen 1994) in the sense that the extraction of FSM may not re ect ....

[Article contains additional citation context not shown here]

Manolios, P., & R. Fanelli. 1994. First order recurrent neural networks and deterministic nite state automata. Neural Computation 6(6):1154{ 1172.

....with A2 solves it within 2797 trials. We also ran another experiment with architecture A2, but without self connections for hidden units. Guessing solved the problem within 250 trials on average. Tomita grammars. Many authors also use Tomita s grammars [30] to test their algorithms. See, e.g. [2, 32, 22, 17, 16]. Since we already tested parity problems above, we now focus on a few parity free Tomita grammars (nr.s #1, #2, #4) Previous work even facilitated the problems by restricting sequence length. e.g. in [17] maximal test (training) sequence length is 15 (10) Reference [17] reports the number ....

P. Manolios and R. Fanelli. First-order recurrent neural networks and deterministic finite state automata. Neural Computation, 6:1155--1173, 1994.

.... strategy that is used in the genetic algorithms can be made more efficient: after a change in the system structure, the local minimum of the continuous cost function can be searched using some traditional technique (see [4] Learning a finite state machine structure by examples, as presented in [5], suggests an approach to iteratively enhancing the network structure also in this more complex case. It is not only the neural networks theory that may benefit of the above result looking merely at the dynamic system formulation (3) it becomes evident that all of the phenomena found in the ....

Manolios, P. and Fanelli, R.: First-Order Recurrent Neural Networks and Deterministic Finite State Automata. Neural Computation 6, 1994, pp. 1155--1173.

.... machines (FSM) a number of researchers set out to test whether DTRNN with real valued, continuous sigmoid activation functions could learn FSM behavior from samples (Cleeremans et al. 1989; Pollack 1991; Giles et al. 1992; Watrous Kuhn 1992; Maskara Noetzel 1992; Sanfeliu Alqu ezar 1994; Manolios Fanelli 1994; Forcada Carrasco 1995; Neco Forcada 1996; Gori et al. 1998) The use of sigmoid functions is motivated by the need to have error functions which are di erentiable with respect to the weights so that gradient descent algorithms may be used for learning. The results of these works show that, ....

.... Once the DTRNN has been trained, researchers use specialized algorithms to extract FSM from the dynamics of the DTRNN; some use a straightforward equipartition of neural state space followed by a branch and bound algorithm (Giles et al. 1992) or a clustering algorithm (Cleeremans et al. 1989; Manolios Fanelli 1994; Gori et al. 1998) Very often, the nite state automaton extracted behaves correctly, and then does so, obviously, for strings of any length. However, automaton extraction algorithms have been criticised (Kolen Pollack 1995; Kolen 1994) in the sense that the extraction of FSM may not re ect ....

[Article contains additional citation context not shown here]

Manolios, P., & R. Fanelli. 1994. First order recurrent neural networks and deterministic nite state automata. Neural Computation 6(6):1154{ 1172.

....Forcada Carrasco, 1995) and a number of approaches to analysing recurrent networks have been developed. One of the principal themes has been the use of clustering and minimization techniques to extract a Finite State Automaton (FSA) that approximates the network s behavior (Giles et al. 1992, Manolios Fanelli, 1994, Tino Sajda, 1995) Casey (1996) showed that if the network robustly models an FSA, the method proposed in (Giles et al. 1992) will successfully extract the FSA given a fine enough resolution. However in many cases the language induced by the network is not regular, and therefore cannot be ....

Manolios, P. & R. Fanelli, 1994. First order recurrent neural networks and deterministic finite state automata, Neural Computation 6(6), 1155--1173.

.... papers also use the socalled parity problem , e.g. Bengio et al. 1994) Bengio and Frasconi (1994) Some of Tomita s grammars (1982) are also often used as benchmark problems for recurrent nets (see, e.g. Bengio and Frasconi, 1995; Watrous and Kuhn, 1992; Pollack, 1991; Miller and Giles, 1993; Manolios and Fanelli, 1994). This paper exemplifies: such problems can be solved more quickly by random weight guessing than by the proposed algorithms. Guessing. With a given architecture, random weight guessing works as follows: REPEAT randomly initialize the weights UNTIL the resulting net happens to classify all ....

....self connections for the hidden units. Guessing solved the problem within 250 trials on average. Tomita grammars. Many authors also use Tomita s grammars (1982) to test their algorithms. See, e.g. Bengio and Frasconi (1995) Watrous and Kuhn (1992) Pollack (1991) Miller and Giles (1993) Manolios and Fanelli (1994). Since we already tested parity problems above, we now focus on a few parity free Tomita grammars (nr.s #1, #2, #4) Previous work facilitated the problems by restricting sequence length. e.g. Miller and Giles maximal test (training) sequence length is 15 (10) Miller and Giles (1993) report ....

Manolios, P. and Fanelli, R. (1994). First-order recurrent neural networks and deterministic finite state automata. Neural Computation, 6:1155--1173.

....to the weighted sum of neuron inputs. Then, the structure is transformed into the space of neuron activations (section 4) There are many applications where oscillatory dynamics of recurrent networks is desirable. For example, when trained to act as a finite state machine ( 7] 8] 9] 10] [14], 16] 17] 18] the network has to induce a stable representation of state transitions associated with each input symbol of the machine. A transition may have a character of a loop (do not move from the current state when the symbol x is presented) or a cycle (when repeatedly presenting the ....

....associated with each input symbol of the machine. A transition may have a character of a loop (do not move from the current state when the symbol x is presented) or a cycle (when repeatedly presenting the same input, we eventually return to the state where we have started) As reported in [6] [14], and [15] loops and cycles associated with an input symbol x are usually represented as attractive fixed points and periodic orbits respectively of the underlying dynamical system corresponding to the input x. It was proved by Hirsh [11] that when all the weights in a recurrent network with ....

P. Manolios and R. Fanelli. First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6(6):1155--1173, 1994.

....state corresponding to switch q 0 that was extracted from the SLRNN. This machine shall be called M1. One of the advantages of using second order SLRNNs is the ease with which automata can be extracted from the the trained or training networks. However, first order SLRNNs could also be used (Manolios Fanelli, 1994; Miller Giles, 1993) Details on the method of SSM extraction that we used can be found in (Giles et al. 1992; Giles et al. 1992) The left column (S) contains the number of each state. State 1 in Tables 2 to 9 corresponds to the initial state. The next column (O) contains the output ....

Manolios, P. & Fanelli, R. (1994). First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6 (6), 1154--1172.

....that were trained without a priori knowledge. Recurrent networks are inherently more powerful than feed forward networks because they are able to dynamically store and use state information indefinitely due to the built in feedback [29] In particular, they can be encoded [20, 18] and trained [8, 11, 16, 26, 31, 33] to behave like deterministic, sequential finite state automata. Methods for inserting prior knowledge into recurrent neural networks have been previously discussed [4, 6, 7, 12, 15, 21] It has been demonstrated [12, 21] that prior knowledge can significantly reduce the amount of training ....

P. Manolios and R. Fanelli, "First order recurrent neural networks and deterministic finite state automata, " Neural Computation, vol. 6, no. 6, pp. 1154--1172, 1994.

....defining fixed points. The most typical bifurcation responsible for the creation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [27] 24] 6] 8] 13] 15] 36] 37] [26], 9] 21] 5] State units activations represent past histories and clusters of these activations can represent the states of the generating automaton [16] In this contribution, the relationship between a RNN and a finite state machine it exactly mimics 1 is investigated from two points of ....

P. Manolios and R. Fanelli. First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6(6):1155--1173, 1994.

....translated into dynamical scenarios in RNN state space. During the training, RNN goes through a sequence of bifurcations leading to a desired attractive behavior. Activations of recurrent neurons group into characteristic clusters corresponding to induced attractive sets inside the RNN state space [19, 3, 17]. It is natural to interpret the activation clusters as significant states the network operates on. By detecting the clusters one describes the behavior of RNN using a deterministic, or stochastic FSM. Contrary to RNN, the knowledge encoded in FSM is amenable to formal analysis using automata ....

P. Manolios and R. Fanelli. First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6(6):1155--1173, 1994.

No context found.

P. Manolios and R. Fanelly, "First order recurrent neural networks and deterministic finite state automata," Neural Computation, vol. 6, pp. 1155--1173, 1994.

No context found.

Manolios, P. & Fanelli, R. (1994). First order recurrent neural networks and deterministic finite state automata. Neural Computation, 6 (6), 1155--1173.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC