N. Alon, A.K. Dewdney, and T.J. Ott, Efficient simulation of finite automata by neural nets, Journal of the ACM 38 (2) (1991) 495--514.  Home/Search   Document Not in Database   Summary   ACM   TOC   Related Articles   Check

....function (4) which have only a finite number of global network states. Thus, their computational power corresponds to that of finite automata [37] and they can shortly be called neuromata. Also finer Energy Based Computation with Symmetric Hopfield Nets 11 descriptive measures have been studied [1] to find out how efficient such neural implementations of finite automata can be. It has been shown that a neuromaton of size O( p q) can be constructed which, for a constant period p = 4 of presenting the input bits, simulates a given deterministic finite automaton with q states [33, 34] and in ....

N. Alon, A.K. Dewdney, and T.J. Ott, Efficient simulation of finite automata by neural nets, Journal of the ACM 38 (2) (1991) 495--514.

....programs. In [41] it has been proven that a recursive neural network made up of neurons using linear saturation activation functions can simulate an universal Turing machine. Other approaches relate recursive neural networks to deterministic finite state machines (for first order networks see [31, 8, 1]; for second order networks see [30, 47, 16] for radial basis function networks see [13] for iterated function systems see [24, 23] But as shown for varying examples (see e.g. 45, 25] constraints on the network architecture may reduce the computational power of the considered network and ....

N. Alon and A. Dewdney and T. Ott. Efficient simulation of finite automata by neural nets. Journal of the Association for Computing Machinery, 38(2), pp. 495--514, 1991.

....formulation naturally requires that the network grow with increasing input size, i.e. that we actually consider nonuniform sequences of networks, one for each input 1 An interesting question here is how efficient is the representation of finite automata as neural nets. It was shown recently in [3] that representing an automaton of n states may require Omega Gammaq n log n) 1=3 ) gates in the worst case. Cyclic nets 10 size. If the only part of the network that changes is the set of input neurons, then this model is computationally equivalent to the sequential input, uniform network ....

Alon, N., Dewdney, A. K., Ott, T. J. Efficient simulation of finite automata by neural nets. J. Assoc. Comp. Mach. 38 (1991), 495--514.

....time. On the other hand, networks with exponentially large weights may indeed require an exponential time to stabilize, as was first shown in [25] for synchronous 1 An interesting question here is how efficient is the representation of finite automata as neural nets. It was shown recently in [3] that representing an automaton of n states may require Omega ( n log n) 1=3 ) neurons in the worst case. updates (a simplified construction appears in [23] and in [28] for a particular asynchronous update rule. For a different update rule the latter result actually follows already from ....

Alon, N., Dewdney, A. K., Ott, T. J. Efficient simulation of finite automata by neural nets. J. Assoc. Comp. Mach. 38 (1991), 495--514.

....It would certainly be interesting to have a better bound. The use of multi tape Turing Machines may reduce the bound. Furthermore, it is quite possible that with some care in the construction one may be able to drastically reduce this estimate. One useful tool here may be the result in [1] applied to the control unit here we used a very inefficient simulation. 9 Non Deterministic Computation A non deterministic processor net is a modification of a deterministic one, obtained by incorporating a guess input line (G) in addition to the validation and data lines. Hence, the ....

N. Alon, A.K. Dewdney, T.J. Ott, "Efficient simulation of finite automata by neural nets," J. A.C.M. 38 (1991): 495-514.

....in Table 1. The methods differ in the choice of the discriminant function (hard limiting, sigmoidal, radial basis function) the size of the constructed network and the restrictions that are imposed on the weight alphabet. The results in [17] improve the upper and lower bounds reported in [1] for DFAs with only two input symbols. Those bounds can be generalized to DFAs with m input symbols. 33 Author(s) Nonlinearity Order # Neurons # Weights Minsky (1967) hard first O(mn) O(mn) Alon (1991) hard first O(n 3=4 ) Frasconi (1993) sigmoid first O(mn) O(n 2 ) Horne (1994) hard ....

N. Alon, A. K. Dewdney, and T. J. Ott. Efficient simulation of finite automata by neural nets. Journal of the Association for Computing Machinery, 38(2):495--514, 1991.

....methods can be computationally limited. There has also been specific work on automata representation in neural networks, see for example, Casey [23] convergence of FSA extraction) Omlin et al. [138] also a comparison of the complexity of many encoding methods and an extension of work by Alon [7]) Frasconi [50] radial basis functions) Maass [122] spiked neurons) This type of work is continuing in other computation structures, examples being graphs and tree grammars (Frasconi et al. [51] Sperduti [187] and is important for broadening the scope and power of neural computing systems. ....

N. Alon, A.K. Dewdney, and T.J. Ott. Efficient simulation of finite automata by neural nets. Journal of the Association for Computing Machinery, 38(2):495--514, 1991.

....neural networks (RNNs) has attracted the attention of researchers for several reasons, ranging from the pursuit of hardware implementations to the integration (and improvement) of symbolic and connectionist approaches to grammatical inference and recognition. Some previous works (Minsky 1967; Alon et al. 1991; Goudreau et al. 1994) have shown how to build different RNN models, using hardlimiting activation functions, that perfectly simulate a given finite state machine. None of these approaches yields the minimum size RNN which is required. Minsky s method (1967) uses a recurrent layer of ....

....method (1967) uses a recurrent layer of McCulloch Pitts units to implement the state transition function, and a second layer of OR gates to cope with the output function; the recurrent layer has n Theta m units, where n is the number of states and m is the number of input symbols. The method by Alon et al. 1991) uses a three layer recurrent network which needs a number of threshold cells of order n 3=4 Theta m. Recently, Goudreau et al. 1994) have proven that, while second order single layer RNNs (SLRNNs) can easily implement any n state automaton using n recurrent units (and a total number of n 2 ....

[Article contains additional citation context not shown here]

Alon, N., Dewdney, A.K., Ott, T.J. 1991. Efficient simulation of finite automata by neural nets.

....was guaranteed to use weight values limited to the set f1; 2g : Since a recurrent neural network with k hard limiting nodes is capable of representing as many as 2 k states, one might wonder if an m state FSM could be implemented by a network with log m nodes. However, it was shown in (Alon et al. 1991) that the node complexity for a standard fully connected network, such as a Hopfield network (Hopfield, 1982) is Omega i (m log m) 1=3 j . They were also able to improve upon Minsky s result by providing a construction which is guaranteed to yield no more than O Gamma m 3=4 Delta ....

....behavior is unique up to a relabelling of the states (Hopcroft and Ullman, 1979) The number, NM , of different minimal FSMs with m states will be used to determine lower bounds on the number of gates required to implement an arbitrary FSM in a recurrent neural network. It has been shown that (Alon et al. 1991) (2m) m (2 m Gamma 2) m NM : However, it will be easier to work with the simpler bound (2m) m NM ; 1) which can be used without affecting the final results. 2.2 Recurrent Neural Networks As pointed out in (Nerrand et al. 1993) many of the most popular discrete time recurrent neural ....

[Article contains additional citation context not shown here]

Alon, N., Dewdney, A., and Ott, T. (1991). Efficient simulation of finite automata by neural nets.

....It 75 would certainly be interesting to have a better bound. The use of multi tape Turing Machines may reduce the bound. Furthermore, it is quite possible that with some care in the construction one may be able to drastically reduce this estimate. One useful tool here may be the result in [ADO91] applied to the control unit here we used a very inefficient simulation. 5.5 Removing the Sigmoid From the Main Level Here, we show how to construct an equivalent network to the above, in which neurons in the main level compute linear combinations only. In the following construction, we ....

N. Alon, A.K. Dewdney, and T.J. Ott. Efficient simulation of finite automata by neural nets. J. A.C.M., 38:495--514, 1991.

....learning of unknown grammars. In figure 2b is a randomly generated minimal 10 state regular grammar created by a program in which the only inputs are the number of states of the unminimized DFA and the alphabet size p. A good estimate of the number of possible unique DFA is (n2 n n pn n ) [Alon, et al. 1991] where n is number of DFA states) The shaded state is the start state, filled and dashed arcs represent 1 and 0 transitions and all final states have a shaded outer circle. This unknown (honestly, we didn t look) DFA was learned with both 6 and 10 hidden state neuron second order recurrent nets ....

N. Alon, A.K. Dewdney, and T.J.Ott, `Efficient Simulation of Finite Automata by Neural Nets, Journal of the ACM, Vol 38, p. 495 (1991).

....are computationally as powerful as any Turing Machine. Note that at least since the classical work of McCulloch and Pitts in the 1940s, it has been clear how to simulate logic gates by networks of threshold (binary valued) neurons, and hence how to obtain finite automata using such nets (see e.g. [1] for more recent work on that problem) One can simulate Turing machines if one allows a potentially unbounded number of neurons; see e.g. 4] for variations on this theme and relations to cellular automata. Since we insist on a fixed number of neurons, which does not increase during the ....

Alon, N., A.K. Dewdney, and T.J. Ott, "Efficient simulation of finite automata by neural nets," J. A.C.M. (1991): to appear.

.... due to the dynamical nature of recurrent networks making predictions about the generalization performance of trained recurrent networks difficult [Zeng et al. 1993] Methods for constructing DFAs in recurrent networks with hard limiting neurons discriminant functions have been proposed [Alon et al. 1991, Horne and Hush, 1994, Minsky, 1967] methods for constructing networks with sigmoidal and radial basis discriminant functions are discussed in [Frasconi et al. 1993, Gori et al. 1994, Giles and Omlin, 1993] We prove that recurrent networks with continuous sigmoidal discriminant functions ....

Alon, N., Dewdney, A., and Ott, T. (1991). Efficient simulation of finite automata by neural nets. Journal of the Association for Computing Machinery, 38(2):495--514.

....height = dlog 2 me 2. The resulting network then has integer weights, with w max = 2, and recognizes the language L with delay : Remark 4.4 One can obtain different noise tolerance vs. delay tradoffs using the recent more advanced simulations of finite automata by threshold logic networks ([Alon, Dewdney, Ott, 1991], Horne, Hush, 1996] Indyk, 1995] For instance, Horne, Hush, 1996] presents a simulation of m state finite automata by threshold logic networks with O( p m log m) units, connection weights Sigma1, and delay 4. Thus, one can in Corollary 4.3 achieve a noise tolerance bound of j = O(1= p ....

N. Alon, A. K. Dewdney, T. J. Ott, Efficient simulation of finite automata by neural nets. J. Assoc. Comput. Mach. 38 (1991), 495--514.

.... Shavlik 1993; Shavlik 1994; Towell et al. 1990] where the prior knowledge is not only incomplete but may also be incorrect [Giles and Omlin 1993; Omlin and Giles 1996a] Methods for constructing DFAs in recurrent networks where neurons have hard limiting discriminant functions have been proposed [Alon et al. 1991; Horne and Hush 1996; Minsky 1967] This paper is concerned with neural network implementations of DFAs where continuous sigmoidal discriminant functions are used. Continuous sigmoids offer other advantages besides their use in gradientbased training algorithms; they also permit analog VLSI ....

....of the discriminant function (hard limiting, sigmoidal, radial basis function) the size of the constructed network and the restrictions that are imposed on the weight alphabet, the neuron fan in and fan out. The results in Horne and Hush [1996] improve the upper and lower bounds reported in Alon et al. 1991] for DFAs with only two input symbols. Those bounds can be generalized to DFAs with m input symbols (B. Horne, personal communication) Among the methods which use continuous discriminant functions, our algorithm uses no more neurons than the best of all methods, and consistently uses fewer ....

[Article contains additional citation context not shown here]

ALON, N., DEWDNEY,A.K.,AND OTT, T. J. 1991. Efficient simulation of finite automata by neural nets, JACM 38, 2 (Apr.), 495--514.

.... oe t as transfer function (a neuron with first order connections can be identified as a threshold gate) There has been significant work done on characterizing the node complexity of neural networks by applying known results from the circuit complexity of linear threshold circuits (see Alon et al. [1], Horne and Hush [35, 36] Siu et al. 68, 67] Next we show how some of these results can easily be lifted to the context of FA and DTA. One might especially interested in deriving upper and lower bounds on the node complexity of the FA such that arbitrary DTA can be implemented by. The line of ....

....applied in our hand crafted constructive proofs) But what is the minimum number of neurons required to implement arbitrary DTA A first naive consideration would lead to O(log m) since m states can be effectively encoded by dlog me neurons. Here we want to follow the way shown by Alon et al. [1] and Horne and Hush [36] in deriving a lower bound for the node complexity of RNN implementations of FSA. Let K(m) be the smallest number such that every DTA with m or less states can be implemented by a FA using K(m) or fewer neurons. Let L(m) be the number of pairwise divergent (see 3 By log ....

[Article contains additional citation context not shown here]

Noga Alon, A.K. Dewdney, and Teunis J. Ott. Efficient Simulation of Finite Automata by Neural Nets. Journal of the ACM, 38(2):495--514, April 1991.

.... This is important when a network is used for domain theory revision [19, 27, 30] where the prior knowledge is not only incomplete, but may also be incorrect [13, 22] Methods for constructing DFAs in recurrent networks where neurons have hard limiting discriminant functions have been proposed [1, 18, 21]. This paper is concerned with neural network implementations of DFAs where continuous sigmoidal discriminant functions are used. Stability of an internal DFA state representation implies that the output of the sigmoidal state neurons assigned to DFA states saturate at high gain; a constructed ....

....5.10.1 If any one of the conditions is violated, then the languages accepted by the constructed network and the given DFA are not identical for an arbitrary distribution of the randomly initialized weights in the interval [ GammaW; W ] 5. 11 Comparison with other Methods Different methods [1, 7, 5, 18, 21] for encoding DFAs with n states and m input symbols in recurrent networks are summarized in table 1. The methods differ in the choice of the discriminant function (hard limiting, sigmoidal, radial basis function) the size of the constructed network and the restrictions that are imposed on the ....

[Article contains additional citation context not shown here]

N. Alon, A. Dewdney, and T. Ott, "Efficient simulation of finite automata by neural nets," Journal of the Association for Computing Machinery, vol. 38, no. 2, pp. 495--514, April 1991.

....neural networks may be viewed as a genJune 20, 1998 SUBMITTED TNN: Gori, Kuchler, Sperduti 3 eralization of the well known recurrent neural networks. There are significant results on the node complexity of recurrent neural network implementations of finite state automata (FSA) Alon et al. [13], Horne Hush [14] Recently, recursive neural networks have been proven to possess the computational power of at least Frontier toRoot Automata (Sperduti [15] Kuchler [16] This machine model for tree processing is known to be a generalization of the FSA concept for sequence processing ....

....it consists of the progressive activation of a set of units according to the magnitude of the processed integer. On the basis of this progressive encoding, it is possible to implement both the transition and output function of the desired FRAO. By using the techniques developed by Alon et al. [13], and Horne Hush [14] we are able to derive a lower bound on the node complexity (in the case when no restrictions are placed on the number of layers) which is tight to the four layers upper bound. The paper is organized as follows. First, we introduce recursive neural networks, tree automata ....

[Article contains additional citation context not shown here]

Noga Alon, A.K. Dewdney, and Teunis J. Ott, "Efficient Simulation of Finite Automata by Neural Nets", Journal of the ACM, vol. 38, no. 2, pp. 495--514, Apr. 1991.

.... a: 3 Network Architecture for Fuzzy Automata Theorem 2. 2 enables us to transform any FFA into a deterministic automaton which computes the same membership function : Sigma [0; 1] Various methods have been proposed for implementing deterministic automata in recurrent neural networks [1, 2, 15, 14, 20, 26, 31, 28]. We use discrete time, second order recurrent neural networks with sigmoidal discriminant functions which update their current state according to the following equations: S (t 1) i = h(ff i (t) 1 1 e Gammaff i (t) ff i (t) b i X j;k W ijk S (t) j I (t) k ; 1) where b i is ....

N. Alon, A. Dewdney, and T. Ott, "Efficient simulation of finite automata by neural nets," Journal of the Association for Computing Machinery, vol. 38, no. 2, pp. 495--514, April 1991.

.... This is important when a network is used for domain theory revision [15, 21] where the prior knowledge is not only incomplete, but may also be incorrect [10, 17] Methods for constructing DFA s in recurrent networks where neurons have hard limiting discriminant functions have been proposed [1, 14, 16]. This paper is concerned with neural network implementations of DFA s where continuous sigmoidal discriminant functions are used. Our method is an alternative to an algorithm for constructing DFA s in recurrent networks with first order weights proposed by Frasconi et al. 4, 5, 6] A short ....

....5.5.1 If either one of the two conditions is violated, then the languages accepted by the constructed network and the given DFA are not identical for an arbitrary distribution of the randomly initialized weights in the interval [ GammaW; W ] 5. 6 Comparison with other Methods Different methods [1, 6, 11, 14, 16] for encoding DFA s with n states and m input symbols in recurrent networks are summarized in table 2. The methods differ in the choice of the discriminant function (hardlimiting, sigmoidal, radial basis function) the size of the constructed network and the restrictions that are author(s) ....

[Article contains additional citation context not shown here]

N. Alon, A. Dewdney, and T. Ott, "Efficient simulation of finite automata by neural nets," Journal of the Association for Computing Machinery, vol. 38, no. 2, pp. 495--514, April 1991.

.... This is important when we use a network for domain theory revision [20, 28, 31] where the prior knowledge is not only incomplete, but may also be incorrect [13, 23] Methods for constructing DFAs in recurrent networks where neurons have hard limiting discriminant functions have been proposed [1, 19, 22]. This paper is concerned with neural network implementations of DFAs where continuous sigmoidal discriminant functions are used. Continuous sigmoids offer other advantages besides their use in gradient based training algorithms; they also permit analog VLSI implementation, the foundations ....

....5.10.1 If any one of the conditions is violated, then the languages accepted by the constructed network and the given DFA are not identical for an arbitrary distribution of the randomly initialized weights in the interval [ GammaW; W ] 5. 11 Comparison with other Methods Different methods [1, 5, 6, 19, 22] for encoding DFAs with n states and m input symbols in recurrent networks are summarized in table 1. The methods differ in the choice of the discriminant function (hard limiting, sigmoidal, radial basis function) the size of the constructed network and the restrictions that are imposed on the ....

[Article contains additional citation context not shown here]

N. Alon, A. Dewdney, and T. Ott, "Efficient simulation of finite automata by neural nets," Journal of the Association for Computing Machinery, vol. 38, no. 2, pp. 495--514, April 1991.

....The difficulty of acceptance of a given language by a neural network (the neural complexity of the language) can be quantified by the minimal number of neurons needed to recognize the language. In the context of mealy machines and threshold networks a similar problem was attacked by Alon et al. [1] and Horne and Hush [21] An attempt to predict the minimal second order RNN size so that the network can learn to accept a given regular language is presented in [31] The predicted numbers of neurons were shown to correlate well with the experimental findings. Essentially, a good starting point ....

N. Alon, A.K. Dewdney, and T.J. Ott. Efficient simulation of finite automata by neural nets. Journal of the Association of Computing Machinery, 38(2):495--514, 1991.

.... 1 and 2 , respectively. Later, in the simulation by a net, the state s will be represented in unary, as a vector of the form (0; 0; 0; 1; 0; 0) For any q 2 C 4 , we write i[q] 0 if q 1 2 1 if q 1 2 ; and: q] 0 if q = 0 1 if q 6= 0 : We think of i[1] as the top of stack and [1] as the empty stack operators, respectively. It can never happen that i[q] 1 while [q] 0; hence the pair (i[q] q] can have only three possible values in f0; 1g 2 . Note that in terms of the base 4 expansion (2) i[q] 0 when a 1 = 1 (or q = 0) and ....

.... in the simulation by a net, the state s will be represented in unary, as a vector of the form (0; 0; 0; 1; 0; 0) For any q 2 C 4 , we write i[q] 0 if q 1 2 1 if q 1 2 ; and: q] 0 if q = 0 1 if q 6= 0 : We think of i[1] as the top of stack and [1] as the empty stack operators, respectively. It can never happen that i[q] 1 while [q] 0; hence the pair (i[q] q] can have only three possible values in f0; 1g 2 . Note that in terms of the base 4 expansion (2) i[q] 0 when a 1 = 1 (or q = 0) and i[q] 1 when a 1 = 3. ....

[Article contains additional citation context not shown here]

Alon, N., A.K. Dewdney, and T.J. Ott, "Efficient simulation of finite automata by neural nets," J. A.C.M. 38 (1991): 495-514.

.... and Pitts showed that networks of neuron like elements are capable of implementing some types of finite state machines (FSMs) 16] Later Minsky showed that any FSM could be mapped into such a network [17] More recently, new results have been developed to improve the efficiency of this mapping [1, 12, 24]. All of these results assume that the nonlinearity used in the network is a hard limiting threshold function. However, when recurrent networks are used adaptively, continuous valued, differentiable nonlinearities are almost always used. Thus, an interesting question is how the computational ....

N. Alon, A.K. Dewdney, and T.J. Ott. Efficient simulation of finite automata by neural nets. Journal of the Association of Computing Machinery, 38(2):495--514, 1991.

....L(M ) if an accepting state is reached after the string x has been read by M . Alternatively, a DFA M can also be considered a generator which generates the regular language L(M ) 2. 2 Network Construction Various methods have been proposed for implementing DFAs in recurrent neural networks [1, 2, 12, 13, 22, 28, 32]. We use discrete time, second order recurrent neural networks with sigmoidal discriminant functions which update their current state according to the following equations: S (t 1) i = h(ff i (t) 1 1 e Gammaff i (t) ff i (t) b i X j;k W ijk S (t) j I (t) k ; 1) where b i is ....

N. Alon, A. Dewdney, and T. Ott, "Efficient simulation of finite automata by neural nets," Journal of the Association for Computing Machinery, vol. 38, no. 2, pp. 495--514, April 1991.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC