Abstract

I present a general taxonomy of neural net architectures for processing time-varying patterns. This taxonomy subsumes many existing architectures in the literature, and points to several promising architectures that have yet to be examined. Any architecture that processes timevarying patterns requires two conceptually distinct components: a short-term memory that holds on to relevant past events and an associator that uses the short-term memory to classify or predict. My taxonomy is based on a characterization of short-term memory models along the dimensions of form, content, and adaptability. Experiments on predicting future values of a financial time series (US dollar--Swiss franc exchange rates) are presented using several alternative memory models. The results of these experiments serve as a baseline against which more sophisticated architectures can be compared. Neural networks have proven to be a promising alternative to traditional techniques for nonlinear temporal prediction t...

Citations

2304	Learning Internal Representations by Error Propagation - Rumelhart, Hinton, et al. - 1986
1313	Finding structure in time - Elman - 1990
537	Neural networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992
368	A Learning Algorithm for Continually Running Fully Recurrent Neural Networks’, Neural computation - Williams, Zipser - 1989
293	Phoneme recognition using time-delay neural networks - Waibel, Hanaazawa, et al. - 1989
241	Principles of neurodynamics - Rosenblatt - 1962
228	Attractor dynamics and parallelism in a connectionist sequential machine - Jordan - 1986
214	Tensor product variable binding and the representation of symbolic structures in connectionist systems - Smolensky - 1990
169	Predicting the future: A connectionist approach,” Int - Weigend, Huberman - 1990
152	Nonlinear Signal Processing using Neural Networks - Lapedes, Farber - 1987
150	Learning state space trajectories in recurrent neural networks - Pearlmutter - 1989
86	Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters - Bridle - 1989
66	A focused back-propagation algorithm for temporal pattern recognition - Mozer - 1989
60	Predicting Sunspots and Exchange Rates with Connectionist Networks - Weigend, Rumelhart, et al. - 1991
54	Synergy of clustering multiple back-propagation networks - Lincoln, Skrzypek - 1990
54	Induction of multiscale temporal structure - Mozer - 1992
51	Improving model selection by nonconvergent methods - Finnoff, Hergert, et al. - 1993
50	Temporal association in asymmetric neural networks - Sompolinsky, Kanter - 1986
46	Discovering the hidden structure of speech - Elman, Zipser - 1987
42	Cortical connections and parallel processing: Structure and function - Ballard - 1986
39	Local feedback multilayered networks - Frasconi, Gori, et al. - 1992
37	Sequential state generation by model neural networks - Kleinfeld - 1986
37	Finite Impulse Response Neural Networks with Applications in Time Series Prediction - Wan - 1993
34	Neural computation by concentrating information in time - Tank, Hopfield - 1987
33	Interactive Processes in Speech Perception: The TRACE Model - McClelland, Elman - 1986
24	Learning unambiguous reduced sequence descriptions - Schmidhuber - 1992
23	Incremental development of complex behaviors through automatic construction of sensory-motor hierarchies - Ring - 1991
22	Second order properties of error surfaces: learning time, generalization - LeCun, Kanter, et al. - 1991
18	The problem of learning long-term dependencies in recurrent networks", invited paper at the - Bengio, Frasconi, et al. - 1993
16	A theory for neural networks with time delays - Vries, Principe
15	A dynamical approach to temporal pattern processing - Stornetta, Hogg, et al. - 1987
14	The tempo 2 algorithm: Adjusting time-delays by supervised learning - Bodenhausen, Waibel - 1991
12	Connected-digit speaker-dependent speech recognition using a neural network with timedelayed connections - Unnikrishnan, Hopfield, et al. - 1991
7	Learning to represent state. Unpublished master's thesis - Bachrach - 1988
6	Learning with delayed reinforcement through attention-driven buffering (Technical Report - Myers - 1990
5	Speaker Independent Speech Recognition with Neural Networks and Speech Knowledge - Bengio, Mori, et al. - 1990
5	Adaptive control of HVAC processes using predictive neural networks (Technical Report - Curtiss, Kreider - 1992
4	Experiments on learning by back propagation (Technical report CMU-CS-86-126 - Plaut, Nowlan - 1986
4	Learning acoustic features from speech data using connectionist networks - Watrous, Shastri - 1987
3	CONCERT: A connectionist composer of erudite tunes - Mozer - 1991
3	Rumelhart (Eds.), Backpropagation: Theory, architectures, and applications - E - 1994
2	Global analysis of parallel analog networks with retarded feedback - Herz - 1991
2	Untitled Diploma Thesis. Institute fuer Informatik, Technische Universitaet Muenchen - Hochreiter - 1991
2	Self-organizing maps and adaptive filters - Hemmen - 1991
2	The utility driven dynamic error propagation netowrk (Tech Report CUED/F-INFENG/TR.1 - Robinson - 1987
2	Continuous history compression. Manuscript in preparation - Schmidhuber, Prelinger - 1993
1	Touretzky (Eds.), Advances in neural information processing systems 3 (pp. 162--168 - Lippmann - 1992
1	Addison-Wesley Publishing Company. Mozer 22 - Reading - 1985