Omlin, C.W., and Giles, C.L. (1996). Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937-972.  Home/Search   Document Details and Download   Summary   ACM   TOC   Related Articles   Check

.... approach to real time processing would be to adopt a parallel distributed processing paradigm (e.g. neural networks) 117] Recurrent neural networks (i.e. output feeds back into the input) have been constructed to map single sequences of equal length from an input language to an output language [48, 103]. This idea of encoding finitestate automata into an RNN has recently been refined into what is called neural transducers [135] To complete the transformation of a linear architecture into a parallel architecture, it would be necessary to introduce multiple paths and associated weights in both ....

C. Omlin and C. Giles, "Constructing deterministic finite-state automata in recurrent neural networks," Journal of the ACM, vol. 43, no. 6, pp. 937--972, Nov. 1996.

....by recurrent networks. However, automata simulations have practical consequences: The constructions lead to effective techniques of automata rule insertion and extraction; moreover, the automaton behavior is even learnable from data as demonstrated in computer simulations. It has been shown in [27], for example, that finite automata can be simulated by recurrent networks of the form f y , being a simple projection, and f being a standard feedforward network. The number of neurons which are sufficient in f is upper bounded by a linear term in m, the number of states of the automaton. ....

C. Omlin and C. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(2), 1996.

....The way to represent an automaton in EFuNN T is adding a recurrent network in EFuNN because it has shown that recurrent network is capable of representing automaton. Quite amount research has been done about representing an automaton by first order and second order recurrent network [7] [8] [9] If the automaton could be embedded in EFuNN, EFuNN would be more powerful in such a way that it could not only recognise current changing pattern (this is due to the function of automata) but also take EFuNN s past few steps behaviours into account for next step prediction ....

C. W. Omlin, C. L. Giles, "Constructing Deterministic Finite-state Automata in Recurrent Neural Networks," Journal of the ACM, Vol. 43(6), pp. 937-972, 1996.

....classification. That means, the same function class is considered when dealing with the LRAAM instead of folding networks, but a more difficult 14 minimization task is to be solved. Furthermore, algorithms which start from some prior knowledge if available for example in form of automata rules [23] highly probably find a small empirical error faster than an algorithm which has to start from scratch because the starting point is closer to an optimum value in the first case. Again, the function class remains the same but the initialization of the training process is more adequate. Actually, ....

....particularly difficult [5, 16] and the same holds for folding networks dealing with very high input trees as well. This makes further investigation of alternative methods of learning necessary as the already mentioned method to start from an appropriate initialized network rather than from scratch [20, 23] or to use appropriate modifications of the architecture or the training algorithm [6, 16] But for all algorithms the above bounds on the number of training samples apply and no algorithm however complicated is able to yield valid generalization with a number of examples independent of the ....

C. Omlin and C. L. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(2), 1996.

....to classical mechanisms like finite automata or Turing machines. It has been shown in the literature that Turing machines can be simulated with networks with various activation functions [21,22,38] In addition, direct simulations of automata are possible which even show some robustness to noise [6,26,30]. A generalization of some of the latter approaches to folding networks can be found in [5] one can relate folding networks to so called tree automata. However, we do not want to go into details concerning this topic. We are mainly interested in approximating and learning functions which do not ....

C. Omlin and C. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(2):937-972, 1996.

....4. h k (X j ) # Ym #q j # Q, # k # # : #(q j , # k ) #m , where h k (A) h(x, u k ) x # A for short (see figure 4) Fig. 4. Correctness of the output function of a FSM as computed by a DTRNN. The production of output #(q j , #k ) #m is illustrated. Omlin and Giles [28] have proposed an algorithm for encoding deterministic finite state automata (DFA, a class of FSM) in second order recurrent neural networks which is based on a study of the fixed points of the sigmoid function. Alquezar and Sanfeliu [29] have generalized Minsky s [4] result to show that DFA may ....

....partial a priori knowledge into the DTRNN before training it through gradient descent. One of Carrasco et al. s [22] constructions is explained here in more detail: the encoding of a Mealy machine in a second order DTRNN; this construction is similar to the one proposed earlier by Omlin and Giles [28]. It uses a one hot encoding for inputs and a one hot interpretation for outputs, and nX = Q state units (DTRNN is in state q i at time t if x i [t] is high and the x j [t] j #= i, are low) Then the initial state is chosen so that x I [0] 1 and all other x i [0] 0. A single layer ....

C. W. Omlin and C. L. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937--972, 1996.

....A set of rules designating a domain theory that is partially correct is integrated into a feed forward neural network. The neural network is trained, thus refining the rules. Furthermore, it is possible to integrate a deterministic finite state automaton in a recurrent neural network (see [27]) Rule extraction for feedforward neural nets can be achieved with the KT [11] which produces propositional logic rules. A novel method for the classification of structures by neural networks has been proposed by Sperduti and Starita [35] The authors propose a generalization of a recurrent ....

.... There are also methods for extraction of propositional rules from a trained multilayered perceptron when considered as a black box (see [37] or at the level of individual neurons (see [31] In a parallel line of research a finite state automaton has been extracted from a trained networks (see [27]) Narazaki in [26] deals with the problem of rule extraction in a novel way: he bases his method on the function approximated by the network, rather than considering connection weights. Finally, a method called TREPAN, whereby propositional rules are extracted from a neural network with the aid ....

Omlin C.W. and Lee Giles C., Constructing Deterministic Finite-State Automata in Recurrent Neural Networks. Journal of ACM, Vol. 43, No. 6, 1996 ,pp.937-972

....that a recursive neural network made up of neurons using linear saturation activation functions can simulate an universal Turing machine. Other approaches relate recursive neural networks to deterministic finite state machines (for first order networks see [31, 8, 1] for second order networks see [30, 47, 16]; for radial basis function networks see [13] for iterated function systems see [24, 23] But as shown for varying examples (see e.g. 45, 25] constraints on the network architecture may reduce the computational power of the considered network and the assessment of the computational power of a ....

C.W. Omlin and C.L. Giles. Constructing deterministic finite--state automata in recurrent neural networks. Journal of the ACM, 45(6), pp. 937--972, 1996.

....of SRNs in learning temporal dependencies. This paper explores some of the reasons that the temporal characteristics of the embedded Reber grammar prove so difficult for SRNs to learn and attempts to develop an extension to the SRN architecture that addresses some of its shortcomings. 2 See Omlin and Giles(1994) and the references therein for a discussion of the application of recurrent networks to the problem of grammar induction. CHAPTER 2 ANALYSIS OF THE PROBLEM A closer look at the two grammars in question shows that in the Reber grammar each input symbol can be predicted solely on the basis of the ....

Omlin, C.W., and Giles, C.L. 1994. Constructing Deterministic Finite-State Automata in Recurrent Neural Networks. Technical Report No. 94-3, Computer Science Department, Rensselaer Polytechnic Institute, Troy, NY.

....for constructing recurrent analog neural nets that are robust against realistic types of analog noise. On the other hand we present a method for constructing feedforward analog neural nets that are robust with regard to analog noise of this type. 1 Introduction A fairly large literature (see [Omlin, Giles, 1996] and the references therein) is devoted to the construction of analog neural nets that recognize regular languages. Any physical realization of the analog computational units of an analog neural net in technological or biological systems is bound to encounter some form of imprecision or analog ....

Omlin, C. W., Giles, C. L. "Constructing deterministic finite-state automata in recurrent neural networks", J. Assoc. Comput. Mach. 43 (1996), 937-- 972.

....of inputs and outputs, from strings to labeled trees. Recently, numerous researchers have approached grammatical inference using adaptive models such as recurrent neural networks [22] 10 or IOHMM s [24] 10 It is well known that recurrent neural networks can simulate any finite state automata [52] as well as any multistack Turing machine in real time [53] However, the ability of representing finite automata is not sufficient to guarantee that a given regular grammar can be actually learned from examples [54] A detailed discussion of the language identification problem on connectionist ....

....is the introduction of symbolic prior knowledge into adaptive structural processors. Several prior knowledge injection algorithms have been introduced for recurrent networks, assuming that the available prior knowledge can be expressed in terms of state transition rules for a finite automaton (see [52] for a very detailed theoretical analysis) Known transitions are injected into the recurrent architecture, while unknown rules are supposed to be filled in by data driven learning. The approach aims to reduce the complexity of learning and to improve generalization. After learning, one can also ....

C. Omlin and C. L. Giles, "Constructing deterministic finite-state automata in recurrent neural networks," J. ACM, vol. 43, no. 6, pp. 937--972, 1996.

.... when dealing with formal languages which lie at the heart of symbol oriented models [8] We will consider a type of recurrent neural network that has initially been explored by Jordan [4] and more recently by Elman [1] and Pollack [6] More specifically, we use a second order recurrent network [2, 5] simplified to having only one parameter, and we show by simulation that its nodes can represent the input to the network in a way that captures the essential structure of our context sensitive language. When RNs are applied to processing languages, the solutions provided by the network are often ....

....by simulation that its nodes can represent the input to the network in a way that captures the essential structure of our context sensitive language. When RNs are applied to processing languages, the solutions provided by the network are often best understood from a dynamical systems perspective [5]. This perspective can sometimes offer new insights and provide new mechanisms for solving tasks that are usually dealt with from a more traditional symbolic framework. From this point of view, the problem we will have to face here is to control the non linear dynamics of the network in such a way ....

[Article contains additional citation context not shown here]

C. Omlin and C.L. Giles. Constructing Deterministic Finite-State Automata in recurrent neural networks. Technical report, Computer Science Department, Rensselaer Polytechnic Institute, Troy, NY 12180, 1994.

.... Neural Computation, 13(1) 1 1 Introduction Our understanding of the computational nature of systems of interconnected neuron like elements has grown steadily over the past decades (Hopfield, 1982; Amit, Gutfreund Sompolinsky, 1987; Hornik, Stinhcombe White, 1989; Siegelmann Sontag, 1992; Omlin Giles, 1996, to mention but a few) The extent to which some of these results apply to systems of neurons in the brain, however, remains uncertain, primarily because the models of the neurons, as well as those of the networks used in such studies, do not sufficiently resemble their biological counterparts. ....

Omlin, C. W., and Giles, C. L. 1996. Constructing deterministic finite-state automata in recurrent neural networks. J. ACM 43, 937-972.

....can be proved for folding networks which guarantee the learnability of standard feed forward networks. 4 Finally one connection should be mentioned: Several investigations show the possibility of modelling finite automata or tree automata, respectively, with recurrent or folding networks [3, 12, 14, 19]. Moreover, the approaches examine the question as to whether automata rules can be inserted into networks or extracted from trained networks [3, 12, 14, 19] Since the dynamics of automata and recurrent networks precisely match, these approaches seem very promising in order to make the behaviour ....

.... Several investigations show the possibility of modelling finite automata or tree automata, respectively, with recurrent or folding networks [3, 12, 14, 19] Moreover, the approaches examine the question as to whether automata rules can be inserted into networks or extracted from trained networks [3, 12, 14, 19]. Since the dynamics of automata and recurrent networks precisely match, these approaches seem very promising in order to make the behaviour of neural networks understandable for humans. ....

C. Omlin and C. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(2):937-972, 1996.

....is usually smaller, which comes at the expense of increased computational complexity. 5. 5 Learning Finite State Automata Within the AI community, research has been conducted on general methods that can reverse engineer (learn) finite state automata based on their input output behavior (see e.g. [3,20,78,66,72,86,87]) Finite state automata (FSAs) are learned by observing the result of sequences of actions. Often, algorithms capable of learning FSAs require a pre given homing sequence, i.e. a sequence that resets the state of the finite state machine (a routine that carries a robot to a unique location) ....

C.W. Omlin and Giles. C.L. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 45(6):937, 1996.

....recursive neural networks either by using first order connections or highorder ones. While available prior knowledge expressed in terms of fuzzy state transition rules are injected into a recursive network, unknown rules are supposed to be filled in by data driven learning. We also review results [11, 12, 6], about how recursive neural networks with continuous sigmoidal functions can be constructed such that the encoded dynamics remains stable indefinitely. ....

C. W. Omlin, C. L. Giles, "Constructing deterministic finite-state automata in recurrent neural networks ", Journal of ACM, vol. 43, no. 2, pp. 937-972, 1996.

....bias is determined by the value of the encoded weights in the network. Encoding of prior knowledge in feed forward networks (Knowledge Based Neural Networks) 8] has been predominantly done using Horn clauses. In recurrent neural networks, prior knowledge in the form of DFAs has been encoded[9]. A network is then trained using either real time recurrent learning (RTRL) 7] or back propagation trough time (BPTT) 10] The advantages to using prior knowledge are thus as follows: 1) The learning performance may lead to a faster convergence to a solution, 2) networks that are trained with ....

Omlin, C.W., Giles, C.: Constructing Deterministic Finite-State Automata in Recurrent Neural Networks. Journal of the ACM 43 no. 6 (1996) 937-972

....here we give a proof that such implementations in sigmoid activation RNNs are stable, i.e. guaranteed to converge to the correct prespecified membership. This proof is based on previous work of stably mapping deterministic finite state automata (DFA) in recurrent neural networks reported in [41]. In contrast to DFA, a set of FFA states can be occupied to varying degrees at any point in time; this fuzzification of states generally reduces the size of the model, and the dynamics of the system being modeled is often more accessible to a direct interpretation. From a control perspective, ....

....be stable for indefinite periods of time. We will demonstrate how the stability analysis for 3 The equivalence of FFA and deterministic acceptors was first discussed in [58] and first used for encoding FFA in [56] 4 For reasons of completeness, we have included the main results from [41] which laid the foundations for this and other papers [59, 56] Thus, by necessity, there is some overlap. 4 neural DFA encodings carries over to and generalizes the analysis of stable neural FFA representations. In high level VLSI design a DFA (actually finite state machines) is often used as ....

[Article contains additional citation context not shown here]

C.W. Omlin and C.L. Giles, "Constructing deterministic finite-state automata in recurrent neural networks," Journal of the ACM, vol. 43, no. 6, pp. 937--972, 1996.

....achieves perfect classification on the strings of length 1,000 for H 6:1. 3.4 Stability of the DFA Representation In this section, we summarize theoretical results which prove that the automaton representation derived from the mapping algorithm is robust. For a complete analysis, please see [23]. The encoding algorithm leads to the following special form of the equation governing the network dynamics: S (t 1) i = h(x; H) 1 1 e H(1 Gamma2x) 2 where x is the input to neuron S i , and H is the weight strength. The proof of stability of the internal DFA representation makes use ....

....fixed points of g( can directly be derived from the fixed points of a standard sigmoidal discriminant, and (2) we can use the same condition of Theorem 3.1 to guarantee the existence of three stable fixed points of the fuzzy sigmoidal discriminant function. Applying the analysis technique from [23] to prove stability of the fuzzy internal representation of FFAs in recurrent neural networks yields the following result: 26 1 2 3 4 a 0.5 b 0.3 b 0.2 a 0.6 a 0.9 b 0.7 a 0.1 b 0.4 1 2 4 5 a 0.5 b 0.3 b 0.2 a 0.6 a 0.9 a 0.1 b 0.4 a 0.5 b 0.7 b 0.3 (a) b) 1 2 3 4 5 6 ....

C.W. Omlin and C.L. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937-- 972, 1996.

....limitations of a local RNN architecture. Giles [61] and Kremer [104] show that certain growing methods can be computationally limited. There has also been specific work on automata representation in neural networks, see for example, Casey [23] convergence of FSA extraction) Omlin et al. [138] (also a comparison of the complexity of many encoding methods and an extension of work by Alon [7] Frasconi [50] radial basis functions) Maass [122] spiked neurons) This type of work is continuing in other computation structures, examples being graphs and tree grammars (Frasconi et al. [51] ....

C.W. Omlin and C.L. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937--972, 1996.

No context found.

Omlin, C.W., and Giles, C.L. (1996). Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937-972.

No context found.

Omlin, C. W. & Giles, C. L. (1996a), `Constructing deterministic finite-state automata in recurrent neural networks', Journal of the ACM 43, 937--972.

No context found.

Omlin, C. W., Giles, C. L. "Constructing deterministic finitestate automata in recurrent neural networks", J. Assoc. Comput. Mach. 43 (1996), 937--972.

No context found.

Omlin, C. W., Giles, C. L. "Constructing deterministic finitestate automata in recurrent neural networks", J. Assoc. Comput. Mach. 43 (1996), 937--972.

No context found.

Omlin C., Giles l., Constructing Deterministic Finite-State Automata in Recurrent Neural Networks, Journal of the ACM, vol. 45, no. 6, p. 937, 1996.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC