Spatiotemporal connectionist networks (STCN's) comprise an important class of neural models that can deal with patterns distributed both in time and space. In this paper, we widen the application domain of the taxonomy for supervised STCN's recently proposed by Kremer (2001) to the unsupervised case. This is possible through a reinterpretation of the state vector as a vector of latent (hidden) variables, as proposed by Meinicke (2000). The goal of this generalized taxonomy is then to provide a nonlinear generative framework for describing unsupervised spatiotemporal networks, making it easier to compare and contrast their representational and operational characteristics. Computational properties, representational issues and learning are also discussed and a number of references to the relevant source publications are provided. It is argued that the proposed approach is simple and more powerful than the previous attempts, from a descriptive and predictive viewpoint. We also discuss the relation of this taxonomy with automata theory and state space modeling, and suggest directions for further work.

Present neural models of classical conditioning all suffer from the same shortcoming: local representation of information (therefore, very precise neural prewiring is necessary). As an alternative we develop two neural models of classical conditioning which rely on distributed representations of information. Both models are of the Hopfield type. In the first model the existence of transmission delays is used to store temporal relations. The second model is based on interactions between spatially separated neural fields. Using tools from statistical mechanics we show that behavioural constraints can be met only if the Hebb rule is extended with inter- or intrasynaptic competition. 2 3 1. Introduction Connectionism has redirected the attention of cognitive scientists to learning and to the neural substrate in which cognitive processes are implemented. Conditioning has become an important field in which ideas from neural networks, behavioural science and neurophysiology are combined. ...

ABSTRACT Cyclic patterns ofmotor neuron activity are involved in the production ofmany rhythmic movements, such as walking, swimming, and scratching. These movements are controlled by neural circuits referred to as central pattern generators (CPGs). Some of these circuits function in the absence of both internal pacemakers and external feedback. We describe an associative neural network model whose dynamic behavior is similar to that of CPGs. The theory predicts the strength of all possible connections between pairs ofneurons on the basis ofthe outputs oftheCPG. It also allows themean operating levels ofthe neurons tobededuced from themeasured synaptic strengthsbetween the pairs of neurons. We apply our theory to theCPG controlling escape swimming in the mollusk Tritonia diomedea. The basic rhythmic behavior is shown to be consistent with a simplified model that approximates neurons as threshold units and slow synaptic responses as elementary time delays. The model we describe may have relevance to other fixed action behaviors, as well as to the learning, recall, and recognition oftemporally ordered information.

There has been much interest in applying noise to feedforward neural networks in order to observe their effect on network performance. We extend these results by introducing and analyzing various methods of injecting synaptic noise into dynamically-driven recurrent networks during training. We present theoretical results which show that applying a controlled amount of noise during training may improve convergence and generalization performance. In addition, we analyze the effects of various noise parameters (additive vs. multiplicative, cumulative vs. non-cumulative, per time step vs. per string) and predict that best overall performance can be achieved by injecting additive noise at each time step. Noise contributes a second-order gradient term to the error function which can be viewed as an anticipatory agent to aid convergence. This term appears to find promising regions of weight space in the beginning stages of training when the training error is large and should improve convergen...

A fully connected recurrent network that is capable of storing, recalling, and generating a pattern sequence, is presented. This network reproduces a memorized sequence by synchronous updating, and can independently adjust the duration of occurrence of each pattern in the sequence. Such a capability is obtained by using a state dependent threshold for each cell, which reflects the characteristics of the neuron refractory period, and by the use of the hyperbolic tangent activation function rather than a hard limit. Computer simulations highlight the capabilities of the proposed architecture. I. Introduction The recognition and generation of temporal sequences is fundamental to a wide range of cognitive processes. A sequence is distinguished not only by its composite patterns and pattern transitions, but also by the duration of each pattern. For example, when a melody suddenly comes to mind, both its notes and meters are recalled. If it is either out of tune or off the beat, the melody ...

Introduction One of the fundamental aspects of human intelligence is the ability to process temporal information (Lashley, 1951). Learning and reproducing temporal se- quences are closely associated with our ability to perceive and generate body movements, speech and language, music, etc. A considerable body of neural network literature is devoted to temporal pattern generation (see Wang, 2001, for a recent review). These models generally treat a temporal pattern as a sequence of discrete patterns, called a temporal sequence. Most of the models are based on either multilayer perceptrons with backpropagation training or the Hopfield model of associative recall. The basic idea for the former class of models is to view a temporal sequence as a set of associations between consecutive com- ponents, and learn these associations as input-output transformations (Jordan, 1986; Elman, 1990; Mozer, 1993). To deal with temporal dependencies beyond consecutive components, part of the input layer i

A neural model for temporal pattern generation is used and analyzed for training with multiple complex sequences in a sequential manner. The network exhibits some degree of interference when new sequences are acquired. It is proven that the model is capable of incrementally learning a finite number of complex sequences. The model is then evaluated with a large set of highly correlated sequences. While the number of intact sequences increases linearly with the number of previously acquired sequences, the amount of retraining due to interference appears to be independent of the size of existing memory. The model is extended to include a chunking network which detects repeated subsequences between and within sequences. The chunking mechanism substantially reduces the amount of retraining in sequential training. Thus, the network investigated here constitutes an effective sequential memory. Various aspects of such a memory are discussed at the end of the paper. 1 The work described in th...