Abstract

In training feed-forward neural networks using the backpropagation algorithm, a sensitivity to the values of the parameters of the algorithm has been observed. In particular, it has been observed that this sensitivity with respect to the values of the parameters, such as the learning rate, plays an important role in the final outcome. In this tutorial paper, we will look at neural networks from a dynamical systems point of view and examine its properties. To this purpose, we collect results regarding chaos theory as well as the backpropagation algorithm and establish a relationship between them. We study in detail as an example the learning of the exclusive OR, an elementary Boolean function. The following conclusions hold for our XOR neural network: no chaos appears for learning rates lower than 5, when chaos occurs, it disappears as learning progresses. For non-chaotic learning rates, the network learns faster than for other learning rates for which chaos occurs.

Citations