K. Doya, "Bifurcations of recurrent neural networks in gradient descent learning," IEEE Transactions on Neural Networks', submitted, 1993.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....for this is the bifurcation behavior of nonlinear dynamical systems, where small changes in a network parameter, such as a weight or net input to a unit, can switch the system between drastically different transfer functions. This has been the focus of recent work in learning in recurrent networks [1], but less is known about the computational implications for networks in which each unit is a nonlinear dynamical system in its own right. Certainly, for realism s sake, one must consider individual units dynamics to be significant for network function [2] Biological nervous systems are networks ....

K. Doya, "Bifurcations of recurrent neural networks in gradient descent learning," IEEE Transactions on Neural Networks', submitted, 1993.

....this is the bifurcation behavior of nonlinear dynamical systems, where small changes in a network parameter, such as a weight or net input to a unit, can switch the system between drastically different transfer functions. This has been the focus of re cent work in learning in recurrent networks [1], but less is known about the computational implications for networks in which each unit is a nonlinear dynamical system in its own right. Certainly, for realism s sake, one must consider individual units dynamics to be significant for network function [2] Biological nervous systems are ....

K. Doya, "Bifurcations of recurrent neural networks in gradient descent learning," IEEE Transactions on Neural Networks', submitted, 1993.

....power available. In the case of recurrent networks, this problem may become very serious when dealing with long sequences, because of the Backpropagation through time of the errors. Moreover, another source of troubles for long sequences is the bifurcation of the learning trajectories [69], commonly found by researchers in experiments on inductive inference of regular grammars. 4.1 Local Minima in Multilayered Neural Networks In this section, we propose some artificial examples in which the associated error surface is populated by local minima or other stationary points. These ....

....the presence of very abrupt changes of the cost, that can be monitored by the gradient instability, is the other feature that makes recurrent network training very hard, particularly for long sequences. This feature is related to the presence of bifurcations in the weight learning trajectory [69]. 4.2.1 Mapping local minima from feedforward to recurrent networks In the previous section, we have seen examples of small problems involving feedforward networks giving rise to local minima in the error surface. One may wonder if these examples can be replicated in the case of recurrent ....

[Article contains additional citation context not shown here]

K. Doya, "Bifurcations of recurrent neural networks in gradient descent learning." Connectionist News Neuroprose, 1993.

....of neurons to inputs with different interspike interval patterns, changes in input frequency, and changes in synapse power. In recurrent ANNs, the effects of dynamics (such as bifurcation behavior) on learning rules must either be eliminated or carefully controlled to achieve the desired results [13]. It is our feeling that a synaptic modification procedure in networks of more realistic elements should take advantage of the individuals dynamics; greater complexity at that level should lead to greater network computational power. To be fair, however, it could certainly be argued that, in ....

K. Doya, "Bifurcations of recurrent neural networks in gradient descent learning," IEEE Transactions on Neural Networks, submitted, 1993.

....This can have an undesirable effect on the training process, since the gradient descent learning may get into trouble. At bifurcations points, the output of a network can change discontinuously with the change of parameters and therefore convergence of gradient descent algorithms is not guaranteed [12]. In the following a possible application of these ideas to the problem of determination of the complexity of language recognition by neural networks will be discussed briefly. Any FSM with binary output alphabet f0; 1g can function as a recognizer of a regular language. A word over the input ....

K. Doya. Bifurcations of recurrent neural networks in gradient descent learning. Submitted to IEEE Transactions on Neural Networks, 1993.

....with such inferential problems, because of their clustered representation of the network states. On the other hand, the main problems that seem to affect the success of such inferential methods is that of gradient vanishing [Bengio et al. 1993] and of bifurcation of the weight space trajectory [Doya, 1993] when learning long term dependencies , no matter what recurrent network is used. In particular, in this paper, we propose using a modular neural network architecture in which the activations of each module are updated with their own rate. The updating rates and the connection of the different ....

.... unconstrained one) III Scheduling the activation updating in modular architecture Learning in recurrent networks may be seriously plagued by the presence of local minima [Bianchini et al. 1994] by information loss due large plateau, and by bifurcations in the weight space learning trajectory [Doya, 1993]. The last two problems are essentially due to the need of dealing with long term dependencies and seem to affect very seriously recent attempts to deal with inductive inference of regular grammars. This has been clearly pointed out by Bengio et al. Bengio et al. 1993] who describe failures of ....

Doya, K. (1993). Bifurcations of Recurrent Neural Networks in Gradient Descent Learning. Connectionist News Neuroprose.

No context found.

Kenji Doya. Bifurcations of Recurrent neural networks in Gradient Decent Learning. Submitted to IEEE Transactions on neural networks, 1994.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC