this paper (based on a different data set) was presented at the 1992 North American Winter Meeting of the Econometric SocietyinNew Orleans, Louisiana.

Gradient descent algorithms in recurrent neural networks can have problems when the network dynamics experience bifurcations in the course of learning. The possible hazards caused by the bifurcations of the network dynamics and the learning equations are investigated. The roles of teacher forcing, preprogramming of network structures, and the approximate learning algorithms are discussed.

this paper was entitled "Strong Convergence of Recursive m-Estimators for Models with Dynamic Latent Variables." 1087 1088 Andrews (1988)). As learning constitutes a form of econometric estimation, it is desirable to develop learning theory in a context that allows for dynamic structure comparable to that permitted by presently available econometric theory. This not only permits sophisticated learning behavior, but also opens the possibility for useful new econometric methods to be obtained from the learning theory

We derive a smoothing regularizer for dynamic network models by requiring robustness in prediction performance to perturbations of the training data. The regularizer can be viewed as a generalization of the first order Tikhonov stabilizer to dynamic models. For two layer networks with recurrent connections described by Y (t) = f \Gamma WY (t \Gamma ø) + V X(t) \Delta ; Z(t) = UY (t) ; the training criterion with the regularizer is D = 1 N N X t=1 jjZ(t) \Gamma Z (\Phi; I(t))jj 2 + ae ø 2 (\Phi) ; where \Phi = fU; V; Wg is the network parameter set, Z(t) are the targets, I(t) = fX(s); s = 1; 2; \Delta \Delta \Delta ; tg represents the current and all historical input information, N is the size of the training data set, ae ø 2 (\Phi) is the regularizer, and is a regularization parameter. The closed-form expression for the regularizer for time-lagged recurrent networks is: ae ø (\Phi) = fljjU jjjjV jj 1 \Gamma fljjW jj h 1 \Gamma e fljjW jj\Gamma1 ø i ; ...

In feedforward networks, signals #ow in only one direction without feedback. Applications in forecasting, signal processing and control require explicit treatment of dynamics. Feedforward networks can accommodate dynamics by including past input and target values in an augmented set of inputs. Amuch richer dynamic representation results from also allowing for internal network feedbacks. These types of network models are called recurrent network models and are used by Jordan #1986# for controlling and learning smooth robot movements, and by Elman #1990# for learning and representing temporal structure in linguistics. In Jordan's network, past values of network output feed back into hidden units; in Elman's network, past values of hidden units feed backinto themselves. The main focus of this study is to investigate the relative forecast performance of the Elman type recurrent network models in comparison to feedforward networks with deterministic and noisy data. The salient p...

An artificial neural network approach for dimension reduction of dynamical systems is proposed and applied to conductance-based neuron models. Networks with bottleneck layers of continuous-time dynamical units could make a 2-dimensional model from the trajectories of the Hodgkin-Huxley model and a 3-dimensional model from the trajectories of a 6-dimensional bursting neuron model. Nullcline analysis of these reduced models revealed the bifurcations of the neuronal dynamics underlying firing and bursting behaviors. 1 Introduction The dynamics of neural membrane potential is best described by conductance-based models, for example, the Hodgkin-Huxley (HH) model (Hodgkin and Huxley 1952) and its derivatives, which are usually fourth or higher order differential equation systems. The lower dimensional models that approximate the behaviors of these higher order models have been proposed for faster simulations, rigorous mathematical analyses, and better intuitive understanding of the d...

Asymptotic behavior of a recurrent neural network changes qualitatively at certain points in the parameter space, which are known as "bifurcation points". At bifurcation points, the output of a network can change discontinuously with the change of parameters and therefore convergence of gradient descent algorithms is not guaranteed. Furthermore, learning equations used for error gradient estimation can be unstable. However, some kinds of bifurcations are inevitable in training a recurrent network as an automaton or an oscillator. Some of the factors underlying successful training of recurrent networks are investigated, such as choice of initial connections, choice of input patterns, teacher forcing, and truncated learning equations. 1 Introduction Recurrent neural networks are expected to have versatile capabilities for modeling and controlling dynamical systems. From the fact that multi-layer neural networks can approximate arbitrary mappings [13], it is easy to show that recurrent n...

Hopfield and others have shown that for a fairly broad class of neural network models there exist Lyapunov functions (which insure that the dynamics of the network will remain tame), so long as the coupling matrix is assumed to be sym- metric. It will be shown that this condition may be replaced by a more easily satisfied condition. I will then present a few other results which further restrict the complexity of the parameter space of some network models.

Researchers in analog computation theory have shown that a recurrent neural network (RNN) can be built to simulate a Turing machine (Pollack, 1987b; Siegelmann & Sontag, 1995). Recently, we showed that it is possible to train RNNs which imple ment some aspects of analog computation theory-namely a network can develop trajectories that count symbols (Wiles & Elman, 1995). But what are the implications for psychological models of sequence processing based on RNNs? As a first step toward answering this question, we investigate an RNN in a psycholinguistically motivated task: predict the next letter in a simple Deterministic Context Free Language that has one level of center-embedding. We demonstrate how the network develops simple coordination between trajectories that enable it to perform limited counting, and in some cases generalize to longer strings. We geometrically identify and analyze several properties relevant for this task, including information loss that results from approaching attractors, divergence in phase space that is used to split states, and difficulty in learning temporal dependencies when the input-output probabilities overlap for different input symbols. 1.

Judea Pearl’s Causal Model is a rich framework that provides deep insight into the nature of causal relations. As yet, however, the Pearl Causal Model (PCM) has not had much impact on economics or econometrics. This may be due in part to the fact that the PCM is not as well suited to analyzing economic structures as might be desired. We o¤er the settable systems framework as an extension of the PCM that embodies features of central interest to economists and econometricians: optimization, equilibrium, and learning. Because these are common features of physical, natural, or social systems, our framework may prove generally useful. In particular, settable systems o¤er a number of advantages relative to the PCM for machine learning. Important distinguishing features of the settable systems framework are its countable dimensionality, its treatment of attributes, the absence of a …xed-point requirement, and the use of partitioning and partition-speci…c response functions to accommodate the behavior of optimizing and interacting agents. A series of closely related machine learning examples and examples from game theory and machine learning with feedback demonstrates limitations of the PCM and motivates the distinguishing features of settable systems.