P. Tino and M. Koteles, "Extracting finite-state representation from recurrent neural networks trained on chaotic symbolic sequences," Neural Computation, vol. 10, pp. 284--302, 1999.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....for predicting online the next component in a symbolic sequence. Arithmetic compression [7] is used to evaluate the quality of the predictor. Di#erent symbolic sequence sources ranging from finite state machines to texts in human language are considered in the experiments. Unlike previous works [3, 11, 13] which performed neural o#ine prediction on the kind of sequences studied here, in this paper we concentrate on online prediction. 2 Prediction with DTRNN We have chosen two classical DTRNN: Elman s simple recurrent network (SRN) 4] and the recurrent error propagation network (REPN) 10] Both ....

....significant. Both architectures, REPN and SRN, give comparable results. The DEKF training algorithm gives CR near those of the [0, 4] gram model, whereas those of RTRL are lower. Chaotic Sequences. We show the results for a symbolic sequence composed of the activation measures of a chaotic laser [13]. Figure 2 shows the CR for this sequence (length 10000, 4) The [0, 4] gram model gives a CR of 2.73. When using RTRL, di#erences between REPN and SRN are bigger than before. Again, the DEKF surpasses the results of RTRL. Anyway, RTRL and the REPN give CR (for nX 4) similar to those of ....

Tino, P., M. Koteles (1999), "Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences", IEEE Transactions on Neural Networks, 10(2), pp. 284--302.

....be explained by the familiar information latching problem in recurrent networks when longer time spans are to be latched [3, 4] We argue that BCM based models correspond to variable memory length Markov models. 1 Introduction In our recent study of hybrid symbolic neural time series modeling [1], we trained the second order RNNs on complex symbolic sequences and then extracted finite state neural based predictive models by using only recurrent part of the trained networks and discarding the non recurrent RNN layers. Loosely speaking, the recurrent part of RNN is used to code histories of ....

....activations of recurrent neurons were randomly generated from a uniform distribution over [ 0.5, 0.5] Thirty epochs were used to train the RNN on the laser sequence and five epochs for the Feigenbaum sequence. For both networks, the initial weights were uniformly generated from the interval of [ 1, 1]. In Fig. 2, we present NNLs for the laser and Feigenbaum test sequences. All presented values are the averages over 10 vector quantization runs in our predictive model construction (see section 4) Also shown are the corresponding standard deviations. 6 Discussion The main theme of this paper ....

[Article contains additional citation context not shown here]

Tino P. and Koteles M. (1999) Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Transactions on Neural Networks 10, 284--302.

....containing the symbol 4 are very rare, which is demonstrated by white regions in CGR(S 2 ) shown in figure 1b. Points in CGR(S 2 ) approximate a noisy Sierpinski triangle [5] Chaos game representations of chaotic symbolic sequences and sequences generated by stochastic automata can be found in [15]. III. Formal definitions Consider a finite alphabet A = f1; 2; Ag. The sets of all finite and infinite sequences over A are denoted by A and A respectively. The set of all sequences consisting of a finite, or an infinite number of symbols from A is then = A [ A . The set ....

....2 and corollary 1 suggest using the geometric representations studied in this paper as illustrative, visual codings of the n block statistical structures in sequences over A. For each n block w 2 A , the cube w(X) is colored according to its probability Pn (w) This approach was taken in [15] to monitor the training process of recurrent neural networks and stochastic machines on chaotic symbolic sequences. At certain stages of the training process the models M were used to generate sequences M (S) of length equal to the length of the training sequence S. Then, the n block ....

P. Tino and M. Koteles, "Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences, " IEEE Transactions on Neural Networks, in press, 1999.

....to perform projection pursuit of the input space into the space of BCM activations [5, 6] The (extended) input to recurrent BCM network consists of the code of the current input symbol and the current network state, which is the activation vector of BCM neurons from the previous time step (Fig. 1a) Recurrent BCM networks (BCM RNN) then perform a time conditional projection pursuit, i.e. organize the state space so that the extended input vectors project into the next states unveiling the most prominent departures from the Gaussian distribution in the extented input. After training with ....

....using normalized negative loglikelihood on unseen continuations of the sequences. 2 Second order BCM RNN with lateral inhibition The output of the ith neuron in the 2nd order BCM RNN comprised of n neurons with lateral inhibition of the constant strength between each pair of neurons (Fig. 1a) is [2] c i (t 1) oe nX j;k h w ijk (t) Gamma X fi 6=i w fijk (t) i d j (t)c k (t) o (1) where oe denotes the sigmoid activation function, i.e. oe(x) 1= 1 exp( Gammax) with the slope . The input symbol at time t is coded by the binary input vector d(t) fd j (t)g. The ....

[Article contains additional citation context not shown here]

Tino P. and Koteles M. (1999) Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Transactions on Neural Networks 10, 284-- 302.

....be explained by the familiar information latching problem in recurrent networks when longer time spans are to be latched [3, 4] We argue that BCM based models correspond to variable memory length Markov models. 1 Introduction In our recent study of hybrid symbolic neural time series modeling [1], we trained the second order RNNs on complex symbolic sequences and then extracted finite state neural based predictive models by using only recurrent part of the trained networks and discarding the non recurrent RNN layers. Loosely speaking, the recurrent part of RNN is used to code histories of ....

....activations of recurrent neurons were randomly generated from a uniform distribution over [ 0.5, 0.5] Thirty epochs were used to train the RNN on the laser sequence and five epochs for the Feigenbaum sequence. For both networks, the initial weights were uniformly generated from the interval of [ 1, 1]. In Fig. 2, we present NNLs for the laser and Feigenbaum test sequences. All presented values are the averages over 10 vector quantization runs in our predictive model construction (see section 4) Also shown are the corresponding standard deviations. 6 Discussion The main theme of this paper ....

[Article contains additional citation context not shown here]

Tino P. and Koteles M. (1999) Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Transactions on Neural Networks 10, 284--302.

....box counting fractal dimension and the information dimension estimates on CBR n;k (S) tend to the sequences topological and metric entropies, respectively, scaled by (log 1 k ) Gamma1 . The chaos n block representation codes the n block suffix structure in the following sense [12] see also [14]: if v 2 A is a suffix of length jvj of a string u = rv, r; u 2 A , then u(X) ae v(X) where v(X) is an N dimensional hypercube of side length k jvj . Hence, the longer is the common suffix shared by two n blocks, the closer the n blocks are mapped in the chaos n block representation ....

P. Tino and M. Koteles. Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Transactions on Neural Networks, (in press), 1999.

....containing the symbol 4 are very rare, which is demonstrated by white regions in CGR(S 2 ) shown in figure 1b. Points in CGR(S 2 ) approximate a noisy Sierpinski triangle [5] Chaos game representations of chaotic symbolic sequences and sequences generated by stochastic automata can be found in [15]. III. Formal definitions Consider a finite alphabet A = f1; 2; Ag. The sets of all finite 2 and infinite sequences over A are denoted by A and A respectively. The set of all sequences consisting of a finite, or an infinite number of symbols from A is then A 1 = A [ A . ....

....2 and corollary 1 suggest using the geometric representations studied in this paper as illustrative, visual codings of the n block statistical structures in sequences over A. For each n block w 2 A n , the cube w(X) is colored according to its probability Pn (w) This approach was taken in [15] to monitor the training process of recurrent neural networks and stochastic machines on chaotic symbolic sequences. At certain stages of the training process the models M were used to generate sequences M (S) of length equal to the length of the training sequence S. Then, the n block ....

P. Tino and M. Koteles, "Extracting finite state representations from recurrent neural networks trained on chaotic symbolic sequences, " IEEE Transactions on Neural Networks, in press, 1999.

No context found.

P. Tino and M. Koteles, "Extracting finite-state representation from recurrent neural networks trained on chaotic symbolic sequences," Neural Computation, vol. 10, pp. 284--302, 1999.

No context found.

Tino,P.&Koteles, M. (1999). Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Transactions on Neural Networks, 10 (2), 284--302.

No context found.

Tino, P. & Koteles, M. (1999), `Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences', IEEE Transactions on Neural Networks 10(2), 284--302.

No context found.

P. Tino and M. Koteles. Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Trans. Neural Networks, 10(2):284--302, 1999.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC