Abstract

Abstract | We survey learning algorithms for recurrent neural networks with hidden units, and put the various techniques into a common framework. We discuss xedpoint learning algorithms, namely recurrent backpropagation and deterministic Boltzmann Machines, and non- xedpoint algorithms, namely backpropagation through time, Elman's history cuto, and Jordan's output feedback architecture. Forward propagation, an online technique that uses adjoint equations, and variations thereof, are also discussed. In many cases, the uni ed presentation leads to generalizations of various sorts. We discuss advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones, continue with some \tricks of the trade" for training, using, and simulating continuous time and recurrent neural networks. We present somesimulations, and at the end, address issues of computational complexity and learning speed.

Citations

2802	Optimization by Simulated Annealing - Kirkpatrick, Gelati, et al. - 1983
2307	Learning internal representations by error propagation - Rumelhart, Hinton, et al. - 1986
1446	A new approach to linear filtering and prediction problems - Kalman - 1960
1314	Finding structure in time - Elman - 1990
448	Adaptive switching circuits - Widrow, Hoff - 1960
440	Learning regular sets from queries and counterexamples - Angluin - 1987
439	Beyond Regression: New Tools for Prediction and Analysis - Werbos
436	Neural” computation of decision optimization problems - Hopfield, Tank - 1985
383	Identification and control of dynamic systems using neural networks - Narendra, Parthasarathy - 1990
368	A Learning Algorithm for Continually Running Fully Recurrent Neural Networks’, Neural computation - Williams, Zipser - 1989
364	A learning algorithm for Boltzmann machines - Ackley, Hinton - 1985
283	Increased rates of convergence through learning rate adaptation - Jacobs
256	Backpropagation applied to handwritten zip code recognition - LeCun, Boser, et al. - 1989
235	Parallel Distributed Processing - Rumelhart, McClelland - 1986
228	Attractor dynamics and parallelism in a connectionist sequential machine - Jordan - 1986
217	Cooperative computation of stereo disparity - Marr, Poggio - 1976
215	Backpropagation through time: what it does and how to do it - Werbos - 2002
211	Absolute stability of global pattern formation and parallel memory storage by competitive neural networks - Cohen, Grossberg
159	Analysis of hidden units in a layered network trained to classify sonar targets. Neural Network - Gorman, Sejnowski - 1988
154	Learning distributed representations of concepts - Hinton - 1986
152	Nonlinear Signal Processing using Neural Networks - Lapedes, Farber - 1987
151	A time-delay neural network architecture for isolated word recognition - Lang, Waibel, et al. - 1990
150	Learning state space trajectories in recurrent neural networks - Pearlmutter - 1989
136	How brains make chaos in order to make sense of the world - Skarda, Freeman - 1987
132	Finite state automata and simple recurrent networks - Cleeremans, Servan-Schreiber, et al.
127	A mean field theory learning algorithm for neural networks - Peterson, Anderson - 1987
125	Generalization by weight–elimination with applications to forecasting - Weigend, Rumelhart, et al. - 1991
124	Generalization of backpropagation to recurrent neural networks - Pineda - 1987
105	An efficient gradient-based algorithm for on-line training of recurrent network trajectories - Williams, Peng - 1990
103	editors. Neural Networks for Control - Sutton, Werbos - 1990
100	D.: Gradient-based learning algorithms for recurrent networks and their computational complexity - Williams, Zipser - 1995
99	Massively Parallel Parsing: A Strongly Interactive Model of Natural Language Interpretation - Waltz, Pollack - 1985
77	Optimal perceptual inference - Hinton, Sejnowksi - 1983
75	Generalization of backpropagation with application to a recurrent gas market model, neural networks - Werbos - 1988
70	Analog Computation Via Neural Networks - Siegelmann, Sontag - 1994
68	A learning rule for asynchronous perceptrons with feedback in a combinatorial environment - Almeida - 1987
66	Simulation of chaotic EEG patterns with a dynamic model of the olfactory system - Freeman - 1987
66	A focused back-propagation algorithm for temporal pattern recognition - Mozer - 1989
65	FIR and IIR synapses, a new neural network architecture for time-series modeling - Back, Tsoi - 1991
64	Learning to control an unstable system with forward modeling - Jordan, Jacobs - 1990
63	Random DFA’s can be approximately learned from sparse uniform examples - LANG - 1992
63	Stationary and nonstationary learning characteristic of the LMSadap99 e filter - Widrow, McCool, et al. - 1976
62	Adapting bias by gradient descent: An incremental version of deltabar-delta - Sutton - 1992
59	Progress in supervised neural networks - Hush, Horne - 1993
58	Gradient methods for the optimization of dynamic systems containing neural networks - Narendra, Parthasarathy
56	Deterministic Boltzmann learning performs steepest descent in weight-space - Hinton - 1990
54	Induction of multiscale temporal structure - Mozer - 1992
54	Learning complex, extended sequences using the principle of history compression - Schmidhuber - 1992
50	Learning algorithms for connectionist networks: Applied gradient methods for nonlinear optimization - Watrous - 1987
42	Neural networks and nonlinear adaptive filtering: Unifying concepts and new algorithms - Nerrand, Roussel-Ragot, et al. - 1993