M. C. Mozer 1993, "Neural net architectures for temporal sequence processing," in Time Series Prediction: Forecasting the Future and Understanding the Past,eds.A.S.WeigendandN.A.Gershenfeld (SFI Studies in the Sciences of Complexity, AddisonWesley) , pp. 243--265.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....is particularly useful in the present data scarcity problem for acute type of drugs. By modifying the actual time series with the proposed scheme, the memory of non zero drug sales is retained for a longer period of time. It is easier to train the neural networks with the modified time series [7]. Another solution to the scarcity problem is to recycle old data. That is, allow the neural network to learn the same training data set many times. In the Medicorp project, we allowed the neural networks to cycle through the data approximately 3000 times. The greatest advantage of recycling is ....

M.C. Mozer, "Neural Net Architectures for Temporal Sequence Processing", Predicting the future and understanding the past (Eds. A. Weigend and N. Gershenfeld), Addison-Wesley, 1993.

....and, moreover, significant theoretical difficulties and benefits can already be found at this level. 3 Recurrent Neural Networks Recurrent networks are a natural tool in any domain where time plays a role, such as speech recognition, control, or time series prediction, to mention just a few [8,9,25,41]. They are also used for the classification of symbolic data such as DNA sequences [31] Turing Capabilities The fact that their inputs and outputs may be sequences suggests the comparison to other mechanisms operating on sequences, such as classical Turing machines. One can consider the internal ....

M. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Predicting the future and understanding the past. Addison-Wesley, 1993.

....vestiges of such patterns for a certain period of time. Hence, an STM model can establish temporal associations between consecutive patterns and reproduce their order of occurrence at the network output. There is a number of STM models within the framework of arti cial neural networks [20], 21] 22] The simplest one, called tapped delay lines, involves a bu er containing the most recent symbols from a sequence. Such a bu er consists of time delays serially connected. These lines convert a temporal sequence into a spatial pattern by concatenating the sequence components through ....

....into a spatial pattern by concatenating the sequence components through a xed size window which slides in time. The concatenated vector is then presented to the network. Tapped delay lines are common in neural network models and form the basis of traditional statistical autoregressive models [20]. For further details on the role of time delays in temporal sequence learning, the readers are referred to Herz [23] The number of time delays de nes the memory depth, i.e. the period of time a pattern remains available in the STM. For instance, four time delays indicate that a particular ....

[Article contains additional citation context not shown here]

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Predicting the Future and Understanding the Past, pages 243-264. AddisonWesley, Redwood City, CA, 1993.

....Folding Networks, Computational Learning Theory, PAC Learning, VC Dimension. 1 Introduction Neural networks are successfully used to deal with recursive data, e.g. for time series prediction, for modeling finite automata, for recognizing DNA sequences, or in system identification and controlling [M2, R, OG, NP, S1, S2]. They can naturally be generalized so that they can handle not only linear data but also trees. The so called folding architecture is used for term classification tasks and can be included in more complex scenarios, e.g. controlling search heuristics in automatic theorem proving [SKG] Recurrent ....

M. Mozer, Neural net architectures for temporal sequence processing, Predicting the future and understanding the past, A. Weigend and N. Gershenfeld (eds.), Addison-Wesley, Reading, MA, 1993, 143-164.

....be used in standard connectionistic methods. Regarding folding networks the encoding is trained simultaneously with some classification of the trees which is to be learned using a modification of back propagation [11] This approach has been used very successfully in several areas of application [7, 11, 21, 22, 25]. The RAAM and LRAAM train the encoding simultaneously with a dual decoding such that the composition yields the identity [24, 28] The classification of the encoded trees is trained separately. Here we focus on the capability of learning with these dynamics in principle. We consider information ....

.... descent method like back propagation through structure or back propagation through time, respectively [11, 30] They have been used successfully in several areas of application including time series prediction, control of search heuristics, and classification of chemical data and graphical objects [7, 22, 25]. A similar mechanism proposed for the processing of structured data is the LRAAM. It is possible to define in analogy to the encoding function g y for any mapping G = G 0 ; G 1 ; G k ) and set Y ae an induced decoding G Y : R k where G Y (t) if t 2 ....

M. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Predicting the future and understanding the past. Addison-Wesley, 1993.

....by a dynamic system but it can be in principal the output of any function on sequences. In practical applications this scenario takes place if structural data, time series or lists, has to be considered and the advantage of recurrent networks to deal with inputs of arbitrary length is used [8, 9]. In fact, recurrent networks can be generalized in a natural way to so called folding networks which take not only sequences but more complex structured objects, labeled trees, as inputs. This is a very promising approach which has applications in classical symbolic areas: term classification and ....

M. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Predicting the future and understanding the past. Addison-Wesley, 1994.

....the mapping which is to be approximated is not presented in a recursive form a priori. In practical applications this scenario takes place if structural data, time series or lists are to be considered and the advantage of recurrent networks to deal with inputs of arbitrary length is used [14]. In fact, recurrent networks can be generalized in a natural way to so called folding networks which take not only sequences but more complex structured objects, labeled trees, as inputs. This is a very promising approach which has applications in classical symbolic areas: term classification and ....

....dimension l is referred to as the encoding dimension of a network. The input neurons number m 1, m l of g are called context neurons since they store the context of an input sequence. In this way recurrent networks are used in speech recognition or time series prediction, for example [3, 12, 14]. In the following, we consider the standard Borel oe algebra and topology on any finite dimensional real vector space which are both obtained as the smallest algebra and topology, respectively, containing the open intervals as a subset. Consequently, mappings between real vector spaces are ....

M. Mozer. Neural net architectures for temporal sequence processing, in: A. Weigend and N. Gershenfeld, eds., Predicting the future and understanding the past (AddisonWesley, Reading, MA, 1993).

....based kana hand alphabet recognition experiments. 1 Introduction The development of techniques to allow more reliable temporal data series prediction, estimation and recognition is very important in many disciplines, like weather prediction, financial analysis, and temporal pattern recognition [11]. There are diverse approaches like Hidden Markov Models and Finite States Machines, which are able to do the modeling of time series. The Neural Network (NN) approach is a natural one due to its inherent capability of extracting the structure present in a data set and its universal approximating ....

M. Mozer, "Neural net architectures for temporal sequence processing", in Predicting the Future and Understanding the Past, A. Weigend and N. Gershenfeld, Eds. Redwood City, CA: Addison-Wesley, 1993.

....and used even to recognise individuals from their gestures, depending on the task demands. 2 The Time Delay RBF Model Dynamic neural networks can be constructed by adding recurrent connections to standard multi layer perceptrons which then form a contextual memory for prediction over time [15, 7, 21]. These partially recurrent neural networks can be trained using backpropagation but there may be problems with stability and very long training sequences when using dynamic representations. Instead, we use a simple Time Delay mechanism in conjunction with an RBF network, which we term a TDRBF ....

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. S. Weigend and N. A. Gershenfeld, editors, Time Series Prediction: Predicting the Future and Understanding the Past, pages 243--264. Addison-Wesley, Redwood City, CA, 1994.

....position has been reached. Potential ambiguities during reproduction of multiple trajectories should be resolved, preferably by the network itself. Most neural models for sequence processing and trajectory planning are supervised models based on standard MLP or dynamic recurrent networks [4] [5], 6] 7] 8] 9] 10] They depend on the network designer to establish the correct temporal association between consecutive trajectory points, as well as to resolve ambiguities. In contrast, an unsupervised network can be used to learn the temporal order of a trajectory in an autonomous ....

....unchanged until the end of the current sequence has been reached. The xed context acts as a kind of a global sequence identi er. Time varying context is a kind of STM which can be implemented in various ways. For convenience, we adopted the simplest STM model, the so called tapped delay lines [5], 19] It consists of a sliding time window over the input sequence, collecting the corresponding samples and concatenating them, successively, into a single pattern vector of higher dimensionality. Time dependence between successive samples is then captured by the order of the concatenated ....

[Article contains additional citation context not shown here]

M. C. Mozer, \Neural net architectures for temporal sequence processing," in Predicting the Future and Understanding the Past, A. Weigend and N. Gershenfeld, Eds., pp. 243-264. Addison-Wesley, Redwood City, CA, 1993.

....6 blobs in the near future. 5 Time Series Pre Processing Time series techniques for predicting the evolution of manyvariables havebeenwell documented in [9] The techniques typically recover relationships between past time sequences and their future consequences. These range from neural networks [15]toHMMs[8] For a neural approach, there exist many different possibilities for processing and representing temporal data. The simplest one is to merely rasterize each time series into avector. The window of 128 samples is placed over the 10.0 2.5 5.0 1.5 . WINDOW LINEAR RAMP RASTERIZE ....

M.C. Mozer. Neural net architectures for temporal sequence processing. In A.S. Weigend and N.A. Gershenfeld, editors, Time Series Prediction, 1993.

....outperformed the linear combiner with the same number of taps. The reason being that the recursive parameter is able to adaptively find in time the local region more relevant for the processing task (we tested this structure for echo cancellation [Palkar and Principe, 1994] time series prediction [Mozer, 1994], and system identification [Motter and Principe, 1994] Gamma Kernels for the CFAR test For image processing the memory depth is equivalent to the spatial scale. So an extension of the gamma kernel to 2D would find the best spatial neighborhood to best meet the processing goals. g k n ( n 1 ....

Mozer, M.C., "Neural net architectures for temporal sequence processing", in Time Series Prediction, Ed. Weigend and Gershenfeld, pp 243-264, Addison Wesley, 1994.

.... Another approach is to try to handle the increased amount of data better by selecting out the novel parts of the sequences [Sch91a] Recurrent networks have also been used to control sequence learning by using the network s internal states to act as memories for suitable tasks [Elm90] Pea89] [Moz94]. Ulbricht [Ulb96] has addressed the variable length of sequences for autonomous agent tasks by getting the network to change its state slowly in response to sequential changes. Schmidhuber [Sch91b] has used subgoals to provide decompositional solutions for action sequences where each subgoal is ....

Michael C. Mozer. Neural Net Architectures for Temporal Sequence Processing, pages 243-264. Addison Wesley Publishing, Redwood City CA, 1994.

....psychology (Montague Sejnowski, 1994) biology (Wallis, 1998) route learning and navigation (Schlkopf Mallot, 1995) and blind source separation (Girolami Fyfe, 1996) It is important to emphasize that two elements are essential for sequence recall. First, a mechanism of short term memory (Mozer, 1993) to enable extraction and storage transitions from one pattern to its successors in the sequence (see Eq. 6) Second, the activation dynamics must be defined to mimic the previously learned sequence by moving through correct sequence of stored states (see Eqs. 3) and (4) This way, the next ....

Mozer, M. C. (1993). Neural net architectures for temporal sequence processing. In: Predicting the Future and Understanding the Past, A. Weigend & N. Gershenfeld (Eds.), Redwood City, CA: Addison-Wesley, 243-264.

....learned for total or partial reproduction of the memorized sequence. Most of the artificial neural network (ANN) models that implement this hypothesis are based on either multilayer perceptron trained with a temporal version of gradient based learning algorithms or based on the Hopfield model (see Mozer, 1993; Wang, 1995; Herz, 1995 and references therein) Nevertheless, it is important to emphasize that self organization play a major role in temporal sequence learning, and specially the field of robot learning has gained relevant contributions. The vast majority of models is involved in either ....

Mozer, M. C. (1993). Neural net architectures for temporal sequence processing. In: Predicting the Future and Understanding the Past, A. Weigend & N. Gershenfeld (Eds.), Redwood City, CA: Addison-Wesley, 243-264.

....suggested by neurophysyological and psyhological experiments that spatial information is represented in the brain as ordered pattern maps [1] 5] This is the reason for us to choose a neural solution, that forms a map like representations. Models for short and long term memory (STM, LTM, see [2] 4][9][16] that imply naturally information about the dynamics of the underlying processes have been particularly considered. Related to the so defined research directions, our approach preserves the idea of constructing a topological representation. Building the local dynamical perceptions that ....

M. Mozer, Neural Net Architectures for Temporal Sequence Processing, in A.S. Weigend, N.A. Gershenfeld (eds) Time Series Prediction: Forecasting the Future and Understanding the Past, pp. 243-264, Addison Wesley Publishing Company, 1994.

....architecture is based on the Gamma memory [15] in which the hidden units perform a computation equivalent to a temporal convolution of their input with a kernel that is a gamma density function. This convolution is computed incrementally. Different forms of short term memory are reviewed in [16]. The main disadvantage of the above two schemes (TDNNs and Gamma memory) is that they do not naturally capture temporal structures which are elastic, i.e. where the delay between two events may vary widely. On the other hand, in more general recurrent networks, the relevant (past) context can be ....

....sequence which can be used to predict the next output is partly fixed by the user (which decides which delays to consider) and partly wired in the weights associated to each delay. However, for different sequences, different delays may be appropriate. According to the terminology introduced in [16], the memory is static in the case of TDNNs but it is adaptive in the case of recurrent networks. A simple example of that problem is illustrated with the minimal task of section 5.1. This task involves latching on a bit of information and being able to retrieve it at an arbitrary later time. 3.1 ....

[Article contains additional citation context not shown here]

M. Mozer, "Neural net architectures for temporal sequence processing," in Predicting the future and understanding the past (A. Weigend and N. Gershenfeld, eds.), Redwood City, CA: Addison-Wesley, 1993.

....used for the outputs that feedback to the input of the network (see Fig. 2) and in the Elman s network [40] Internally: Inside each neuron. The latter approach brings us to the so called locally recurrent neural networks (LRNN s) or local feedback multilayer networks (LF MLN) 4] 18] 20] [58]. In these structures, classical infinite impulse response (IIR) linear filters [13] here called also autoregressive moving average (ARMA) models, are used either directly or with some modifications. Different architectures arise depending on how the ARMA model is included in the network. The ....

....units is restricted to the first layer only. Another version of locally recurrent neural network was presented in [54] with a biological motivation: a multilayer connection of perceptrons with low pass temporal filtering of the activation. The major advantages [4] 20] 21] 30] 40] 41] [58] of locally recurrent neural networks with respect to buffered MLP s or fully recurrent networks can be summarized as follows: 1) well known neuron interconnection topology, i.e. the efficient and hierarchic multilayer; 2) small number of neurons required for a given problem, due to the use of ....

M. C. Mozer, "Neural net architectures for temporal sequence processing, " Predicting the Future Understanding the Past, A. Weigend and N. Gershenfeld, Eds. San Mateo, CA: Addison-Wesley, 1993.

.... l identification avec l a aussi de bons r esultats [20, 56, 65] Les mod eles connexionnistes du temps Parall element a ce courant principal d applications une autre question a mobilis e la communaut e connexionniste : quelle est la meilleure architecture pour mod eliser un syst eme dynamique [55] Evidemment tout d epend du probl eme. Dans une revue sur 14 le sujet [19] les auteurs proposent de distinguer trois m ecanismes diff erents permettant de traiter les informations temporelles correspondant a trois niveaux de complexit e diff erents. Le temps peut etre trait e ....

M. C. Mozer, Neural net architectures for temporal sequence processings, in Time series prediction: Forecasting the future and understanding the past, A. S. Weigend and N. A. Gershenfeld, eds., Addison Wesley, Reading, MA, 1993, pp. 243--264.

....example, a common practice is to separate the recurrent training into a recurrent memory network followed by a feedforward association network. Then one can concentrate on the form of the memory adequate for the particular task at hand. Mozer gives a comprehensive discussion of this approach in [Moz93] This is more ad hoc than our approach, and has the disadvantage that it is still very much a trial and error process to determine which form of recurrence memory is better for which type of tasks. Of course, in particular tasks where the network is carefully matched to the problem, this may ....

M.C. Mozer. Neural net architectures for temporal sequence processing. In A.S. Weigend and N.A. Gershenfeld, editors, Predicting the future and understanding the past: a comparison of approaches. Addison-Wesley, Redwood City, 1993.

....radial Gaussian functions for each hidden unit, which simulate the effect of overlapping and locally tuned receptive fields. Dynamic neural networks can be constructed by adding recurrent connections to standard multi layer perceptrons which then form a contextual memory for prediction over time [16, 8, 23]. These partially recurrent neural networks can be trained using back propagation but there may be problems with stability and very long training sequences when using dynamic representations. An alternative is the Time Delay Neural Network (TDNN) model (for an introduction, see [10] which ....

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. S. Weigend and N. A. Gershenfeld, editors, Time Series Prediction: Predicting the Future and Understanding the Past, pages 243--264. Addison-Wesley, Redwood City, CA, 1994.

....hybrid symbolic neural systems may be very useful. 2 Various Forms of Hybrid Neural Architectures Various classification schemes of hybrid systems have been proposed [77, 76, 89, 47] Other characterizations of architectures covered specific neural architectures, for instance recurrent networks [38, 52], or they covered expert systems knowledge based systems [49, 29, 75] Essentially, a continuum of hybrid neural architectures emerges which contains neural and symbolic knowledge to various degrees. However, as a first introduction to the field, we present a simplified taxonomy here: unified ....

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Time series prediction: Forecasting the future and understanding the past, pages 243--264. Addison-Wesley, Redwood City, CA, 1993.

....the Past Sequence 5 Time Series Pre Processing Time series techniques for predicting the evolution of many variables have been well documented in [9] The techniques typically recover relationships between past time sequences and their future consequences. These range from neural networks [15] to HMMs [8] For a neural approach, there exist many different possibilities for processing and representing temporal data. The simplest one is to merely rasterize each time series into a vector. The window of 128 samples is placed over the time series data and the 30 different streams are ....

M.C. Mozer. Neural net architectures for temporal sequence processing. In A.S. Weigend and N.A. Gershenfeld, editors, Time Series Prediction, 1993.

....radial Gaussian functions for each hidden unit, which simulate the effect of overlapping and locally tuned receptive fields. Dynamic neural networks can be constructed by adding recurrent connections to standard multi layer perceptrons which then form a contextual memory for prediction over time [16, 8, 24]. These partially recurrent neural networks can be trained using back propagation but there may be problems with stability and very long training sequences when using dynamic representations. An alternative is the Time Delay Neural Network (TDNN) model (for an introduction, see [10] which ....

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. S. Weigend and N. A. Gershenfeld, editors, Time Series Prediction: Predicting the Future and Understanding the Past, pages 243--264. Addison-Wesley, Redwood City, CA, 1994.

....in particular detail and the above probabilistic formalisms will be employed in deriving a machine learning system for our purposes. More related learning applications include the analysis of temporal phenomena (after all, behaviour is a time varying process) The Santa Fe competition [20] 18] [43] was a landmark project in this subarea of learning. The methods investigated therein employed a variety of learning techniques to specifically model and simulate the behaviours and dynamics of time series data. The phenomena considered ranged from physiological data to J.S. Bach s last unfinished ....

M. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Time Series Prediction, 1993.

....network. Once the network has learned, the network may be considered a recurrent network when used as an iterated prediction network. Here, the output is fed back to the input and the inputs are shifted left (see Figure 5. A thorough discussion of prediction network architecture can be found in [11]) An issue with prediction networks is that the time window (the delayed inputs) parameters must be chosen by the experimenter. With the appearance of recurrent learning algorithms such as RTRL, many hoped that the network would learn to extract the relevant temporal structures from the teacher ....

Mozer, M. C., 1993, Neural net architectures for temporal sequence processing, To appear in: A. S. Weigend and N. A. Gershenfeld (Eds.), Predicting the future and understanding the past: a comparison of approaches, Redwood City: Addison-Wesley. 17

....the fact that the learning method we proposed is bottomup, starting from a fixed segmentation process of the information to an associative learning process. Interesting would also be to investigate a bottom up bottom mechanism, as proposed e.g. by (Grossberg Merrill, 1992) and discussed by (Mozer, 1993), where feedback from the associative memory can activate a tuning mechanism of the threshold parameters of the event recognition modules. Robot s grounding of perceptions and actions One of our requirements for starting this work was that the system should learn quickly, i.e. that it would not ....

Mozer, M.C. (1993). Neural net architecture for temporal sequence processing. In N. Gershenfelds, & A. Weigend (Eds.), Predicting the future and understanding the past. Berkeley, CA: Addison-Wesley Publishing.

....pattern recognition (Port, 1990) although it surely is useful for spatial localization with binaural inputs ( 10 t t t t Input Figure 2: Delay line memory. A delay line tapped at various positions transforms temporal data into a spatial array. 3. 1 Mozer s Typology of Memory Mechanisms Mozer (1993) recently proposed basing a taxonomy of network architectures for temporal pattern processing on three dimensions by which memory in networks can be described. We should consider whether his typology will suit our needs. It our opinion it does not. His three dimensions are the form, content, and ....

....is lost from the array. Mozer points out that a huge variety of forms of memory can be formally realized via the convolution of the vectors to be remembered with an arbitrary kernel function c i (t) x i (t) t X =1 c i (t Gamma )x( where the x( are the input vectors to be remembered (Mozer, 1993). This formulation 11 isolates the mechanism that links time steps, and accommodates delay line memory as well as a memory composed of exponentially decaying unit activations. The recursive update equations for unit activations combine time delays and self recurrent terms. While Mozer points out ....

Mozer, M. C. (1993). Neural net architectures for temporal sequence processing. In Weigend, A. and Gershenfeld, N., editors, Predicting the Future and Understaning the Past, Redwood City, California. Addison-Wesley.

....in the same type of models, applying them to identification with similar results [15, 56, 71] 3.1. 2 Connectionnists models of time Parallel to this main trend of application, another question has mobilized the connectionist community: what is the best architecture for modeling a dynamic system [54] Obviously all depends on the problem. In an article dealing with this subject [13] the authors suggest distinguishing three different mechanisms allowing the treatment of time informations corresponding to three different levels of complexity of the time effect. Time can be treated either from ....

M. C. Mozer, Neural net architectures for temporal sequence processings, in Time series prediction: Forecasting the future and understanding the past, A. S. Weigend and N. A. Gershenfeld, eds., Addison Wesley, Reading, MA, 1993, pp. 243--264.

....proofed to be universal function approximators, this does not necessarily imply their usability in TSP. Traditional way of using neural networks in TSP is to convert the temporal sequence into concatenated vector via a tapped delay line, and to feed the resulting vector as an input to a network [11]. This time delay neural network approach, however, has its well know drawbacks, one of the most serious ones of being the difficulty to determine the proper length for the delay line. Therefore a number of dynamic neural networks models have been designed for TSP to capture inherently the ....

....sequence without the need of external time delay mechanics. In these models learning equations are often described by differential or difference equations and the interconnections between the network units may include a set of feedback connections, i.e. the networks are recurrent in nature (see [11, 14]) Most recurrent neural networks are trained via supervised learning rules. Only quite rare unsupervised neural networks models have been proposed for TSP, although, it can be argued that in temporal sequence analysis unsupervised neural networks could reveal useful information from the temporal ....

M. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Time Series Prediction: Forecasting the Future and Understanding the Past, pages 243--264. Addison-Wesley, 1993.

....so it can be trained with standard backpropagation. There are learning algorithms for recurrent networks with variable strength feedback connections (e.g. Williams and Zipser 1989) but they tend to be computationally expensive. There are also more powerful recurrent architectures (see Mozer 1993 for a review) but they have not been used in language models. 1 This was before backpropagation (Rumelhart, Hinton, and Williams 1986) so there was no known way to adjust the weights of a hidden layer. 8 Elman (1990) showed that an SRN could learn to identify low level constituents in a ....

....really a surprise that the model does not generalize as well as people do. The generalization performance of connectionist models depends heavily on the network architecture used. The sensory and motor networks use a very simple architecture that is known to have limited generalization abilities (Mozer 1993). The human neocortex is much more complex. Presumably, people generalize so well because the architecture that emerges from all of this complexity has very powerful generalization abilities. If so, then reproducing human level generalization is really beyond the scope of this model. The purpose ....

Mozer, Michael C. (1993). Neural net architectures for temporal sequence processing. Preprint. To appear in Predicting the future and understanding the past, eds. A. Weigend and N. Gershenfeld. Redwood City, CA: Addison-Wesley.

....train. Even once a network has learned to emulate the behavior of the system that produced the time series, it is a challenge to extract from the network solution properties that characterize the system, such as the number of degrees of freedom, the degree of nonlinearity, or the amount of noise. Mozer (1993) gives an overview that unifies connectionist approaches to temporal sequence processing. 2.2 Reinforcement Learning In some cases of learning, there is no teacher who can provide all of the desired output values. This forces us to leave the fairly safe ground of supervised learning. In ....

MOZER, Michael C. Neural net architectures for temporal sequence processing. In Weigend and Gershenfeld, eds. (1993).

.... [33] are interesting in the context of state spaces, because they can be derived from Markov models (Barnard, 3] Interesting research on merging logic and connectionist systems is reported in (Sun, 28] Work on processing temporal sequences (a general taxonomy can be found in (Mozer, [21]) is concerned mostly with recognition of temporal patterns, but such systems can also be used to generate sequences (e.g. predict the future) A report on aspects of planning with weightless neural networks (Mrsic Fogel, 23] presents an approach that has its roots in switching theory rather ....

Mozer, M. C. (1993), Neural net architectures for temporal sequence processing, in Weigend, A. and Gershenfeld, N. (Eds.) Predicting the future and understanding the past, Addison-Wesley Publishing, Redwood City CA.

....hybrid symbolic neural systems may be very useful. 2 Various Forms of Hybrid Neural Architectures Various classi cation schemes of hybrid systems have been proposed [77, 76, 89, 47] Other characterizations of architectures covered speci c neural architectures, for instance recurrent networks [38, 52], or they covered expert systems knowledge based systems [49, 29, 75] Essentially, a continuum of hybrid neural architectures emerges which contains neural and symbolic knowledge to various degrees. However, as a rst introduction to the eld, we present a simpli ed taxonomy here: uni ed neural ....

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Time series prediction: Forecasting the future and understanding the past, pages 243-264. Addison-Wesley, Redwood City, CA, 1993.

....suggested by neurophysyological and psyhological experiments that spatial information is represented in the brain as ordered pattern maps [1] 5] This is the reason for us to choose a neural solution, that forms a map like representations. Models for short and long term memory (STM, LTM, see [2] 5][12][20] that imply naturally information about the dynamics of the underlying processes have been particularly considered. Related to the so defined research directions, our approach preserves the idea of constructing a topological representation, for instance in a form of a dynamical graph ) as ....

M. Mozer, Neural Net Architectures for Temporal Sequence Processing, in A.S. Weigend, N.A. Gershenfeld (eds) Time Series Prediction: Forecasting the Future and Understanding the Past, pp. 243-264, Addison Wesley Publishing Company, 1994.

....to the two known objects. The two objects are thereby separated in the temporal domain. 3. 3 Learning and recognition of spatio temporal patterns A crucial issue for the processing of temporal patterns is the choice of a suitable short term memory (for a discussion in the context of BPN see [20]) Short term memories (STM) can be characterized by their temporal resolution and depth. Temporal resolution refers to the degree to which information concerning the elements of the temporal pattern is preserved. Depth refers how far back into the past the memory stores information. The often ....

M. C. Mozer, "Neural Net Architectures for Temporal Sequence Processing", in: Predicting the Future and Understanding the Past, A. Weigend and N. Gershenfeld (Eds.), Addison-Wesley, 1994.

....proofed to be universal function approximators, this does not necessarily imply their usability in TSP. Traditional way of using neural networks in TSP is to convert the temporal sequence into concatenated vector via a tapped delay line, and to feed the resulting vector as an input to a network [11]. This time delay neural network approach, however, has its well know drawbacks, one of the most serious ones of being the difficulty to determine the proper length for the delay line. Therefore a number of dynamic neural networks models have been designed for TSP to capture inherently the ....

....sequence without the need of external time delay mechanics. In these models learning equations are often described by differential or difference equations and the interconnections between the network units may include a set of feedback connections, i.e. the networks are recurrent in nature (see [11, 14]) Most recurrent neural networks are trained via supervised learning rules. Only quite rare unsupervised neural networks models have been proposed for TSP, although, it can be argued that in temporal sequence analysis unsupervised neural networks could reveal useful information from the ....

M. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Gershenfeld, editors, Time Series Prediction: Forecasting the Future and Understanding the Past, pages 243--264. Addison-Wesley, 1993.

....in static patterns, the temporal dimension has to be supplied in an appropriate way. 47] distinguishes the following mechanisms: ffl layer delay without feedback (or time windows) ffl layer delay with feedback ffl unit delay without feedback ffl unit delay with feedback (self recurrent loops) [33] bases his overview on a distinction concerning the type of memory: delay (akin to time windows and delays) exponential (akin to recurrent connections) and gamma (a memory model for continuous time domains) I would like to give a slightly different overview. Given the above discussion of time ....

....Standard learning algorithms like backpropagation, although easy to apply, can cause problems or lead to non optimal solutions. Finally, this type of recurrent net al..so cannot really deal with an arbitrarily long history, for similar reasons as above (see, for instance, 2] cited in [3] or [33]) Examples of applications with Elman networks are [21, 15, 17] copy MLP or RBFN MLP or RBFN Figure 7: An extension of the Elman network as realization of a non linear state space model As hinted upon above, a general non linear version of the state space model is conceivable, as well. By ....

Mozer M.C.: Neural Net Architectures for Temporal Sequence Processing, Predicting the Future and Understanding the Past, in A. Weigend and N. Gershenfeld (Eds.): Time Series Prediction: Forecasting the Future and Understanding the Past, Addison-Wesley Publishing, Redwood City, CA, 1993.

....a large number of approaches to handling sequences have been developed and tested. Sequence processing requires a method for saving information for subsequent time steps. Detailed overviews of neural networks for handling temporal aspects are given, for instance, in [Ulbricht et al. 1992] [Mozer, 1993], Rohwer, 1994] and [Chappelier and Grumbach, 1994] Here the following four methods are distinguished and briefly addressed: 1. Layer delay without feedback 2. Layer delay with feedback 3. Unit delay without feedback 4. Unit delay with feedback These methods can be employed in different layers ....

....scheme for sequence handling methods is, in some parts, comparable to other taxonomies found in literature. The two categorization schemes provided by [Catfolis, 1994] and [Chappelier and Grumbach, 1994] refer mainly to the methods for sequence handling, whereas the scheme presented in [Mozer, 1993] covers also the trainability of network components. 1. Layer Delay without Feedback The most straightforward approach is to use an input window which holds a restricted small number of past sequence elements. Then this part of the time series is analyzed before the window is shifted further in ....

[Article contains additional citation context not shown here]

M.C. Mozer. Neural Net Architectures for Temporal Sequence Processing. In A. Weigend and N. Gershenfeld, editors, Predicting the Future and Understanding the Past. Addison-Wesley Publishing, Redwood City, CA, 1993.

....well as other techniques in a majority of cases. For example, the results of the Santa Fe competition on time series prediction [11] suggest that the performance of neural networks is better than that of other techniques for predicting the future trends in stock prices. A paper authored by Mozer [9], which explains the details of the neural network architectures used in that competition, served as a starting point for the exploration of different neural network architectures. The problem of predicting future sales, as with other time series prediction problems, requires the network to ....

....data to make future predictions. Of the neural network architectures with state, we decided to choose from either recurrent neural networks or time delay neural networks because they seemed to be the most well studied, with a large body of work describing how to set parameters etc. The results in [9] indicate that the predictive performance of recurrent neural networks and time delay networks do not differ greatly. We chose the recurrent neural network architecture because the length of the delay in time delay networks has to be set in advance [7] and because recurrent neural networks are ....

[Article contains additional citation context not shown here]

Mozer, M.C., "Neural Net Architectures for Temporal Sequence Processing," Predicting the future and understanding the past (Eds. A. Weigend and N. Gershenfeld), Addison-Wesley, 1993.

....dollar Deutsch mark rate using a feedforward neural network over daily data. This system takes as input past measurements for this exchange rate, volatility and trend measures derived exclusively from the exchange rate series, and the current and past exchange rates for other currencies. Mozer [5] uses a recurrent neural network that takes tick data and predicts the exchange rate 1, 15, and 60 minutes in the future. It is also important to note that several researchers have used a similar model to our Markov chain model, but at a much coarser resolution. In particular, Engel and Hamilton ....

M.C. Mozer. Neural net architectures for temporal sequence processing. In A.S. Weigend and N.A. Gershenfeld, editors, Time Series Prediction:Forecasting the Future and Understanding the Past, pages 243--64, 1993.

....20] One way of doing without the fixed size buffer while still giving the network a memory is through the provision of recurrent connections among units, so that the network state is a function not only of present external input, but also of past states. Many such architectures have been proposed [18, 6, 13]. Much work has addressed its concern primarily to the issue of the complexity (in terms of the generating string grammar) of the symbolic sequence that can be predicted or recognized by a network [25, 5, 9] Less work with recurrent networks has examined the ways in which other kinds of temporal ....

M. C. Mozer, "Neural net architectures for temporal sequence processing," in Predicting the Future and Understanding the Past (A. Weigend and N. Gershenfeld, eds.), Addison- Wesley Publishing, 1993.

No context found.

M. C. Mozer 1993, "Neural net architectures for temporal sequence processing," in Time Series Prediction: Forecasting the Future and Understanding the Past,eds.A.S.WeigendandN.A.Gershenfeld (SFI Studies in the Sciences of Complexity, AddisonWesley) , pp. 243--265.

No context found.

M. C. Mozer. Neural net architectures for temporal sequence processing. In A. Weigend and N. Ger- 7 shenfeld, editors, Predicting the Future and Understanding the Past, pages 243-264. Addison-Wesley, Redwood City, CA, 1993.

No context found.

Mozer M. C., "Neural Net Architectures for Temporal Sequence Processing", In Proceedings of the NATO Advanced Research Workshop on Time Series Prediction and Analysis, ed. Addison-Wesley, Santa Fe, New Mexico, 1994.

No context found.

M. C. Mozer, "Neural Net Architectures for Temporal Sequence Processing," in Predicting the Future and Understanding the Past, A. Weigend and N. Gershenfeld, Eds. redwood City, CA: Addison-Wesley.

No context found.

M.C. Mozer. Neural net architectures for temporal sequence processing. In A.S. Weigend and N.A. Gershenfeld, editors, Predicting the future and understanding the past: a comparison of approaches. Addison-Wesley, 1993.

No context found.

M. C. Mozer 1993, "Neural net architectures for temporal sequence processing," in Time Series Prediction: Forecasting the Future and Understanding the Past,eds.A.S.WeigendandN.A.Gershenfeld (SFI Studies in the Sciences of Complexity, AddisonWesley) , pp. 243--265.

No context found.

Mozer, M. C. (1993). Neural net architectures for temporal sequences processing. In Weigend, A. S. and Gershenfeld, N. A., editors, Time series prediction: Forecasting the future and understanding the past, volume 15, pages 243-264. Addison Wesley, Reading, MA.

No context found.

Mozer, Michael C.(1993). Neural Net Architectures for Temporal Sequence Processing. In: A. Weigend & N. Gershenfeld(Eds.), Predicting the Future and Understanding 24 the Past, Addison-Wesley Publ., Redwood City, CA.

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC