A self-organizing neural network for sequence classification called SARDNET is described and analyzed experimentally. SARDNET extends the Kohonen Feature Map architecture with activation retention and decay in order to create unique distributed response patterns for different sequences. SARDNET yields extremely dense yet descriptive representations of sequential input in very few training iterations. The network has proven successful on mapping arbitrary sequences of binary and real numbers, as well as phonemic representations of English words. Potential applications include isolated spoken word recognition and cognitive science models of sequence processing. 1 INTRODUCTION While neural networks have proved a good tool for processing static patterns, classifying sequential information has remained a challenging task. The problem involves recognizing patterns in a time series of vectors, which requires forming a good internal representation for the sequences. Several researchers have p...

Abstract not found

In this paper a novelty filter is introduced which allows a robot operating in an unstructured environment to produce a self-organised model of its surroundings and to detect deviations from the learned model. The environment is perceived using the robot’s 16 sonar sensors. The algorithm produces a novelty measure for each sensor scan relative to the model it has learned. This means that it highlights stimuli which have not been previously experienced. The novelty filter proposed uses a model of habituation. Habituation is a decrement in behavioural response when a stimulus is presented repeatedly. Robot experiments are presented which demonstrate the reliable operation of the filter in a number of environments. 1.

Vector quantization consists in nding a discrete approximation of a continuous input. One of the most popular neural algorithms related to vector quantization is the, so called, Kohonen map. In this paper we generalize vector quantization to temporal data, introducing context quantization. We propose a recurrent network inspired by the Kohonen map, the Contextual Self-Organizing Map, that develops near-optimal representations of context. We demonstrate quantitatively that this algorithm shows better performance than the other neural methods proposed so far. 1. Introduction The temporal context present in sequential data is crucial for sequence processing and prediction of future events. An element of a sequence is context-dependent when it cannot be predicted from only one previous element, but also needs additional information provided by the preceding inputs [13]. Most neural techniques for sequence learning are based on recurrent networks, where the context is represented by the ...

Abstract not found

This paper describes an unsupervised neural network model for learning and recall of temporal patterns. The model comprises two groups of synaptic weights, named competitive feedforward and Hebbian feedback, which are responsible for encoding the static and temporal features of the sequence respectively. Three additional mechanisms allow the network to deal with complex sequences: context units, a neuron commitment function, and redundancy in the representation of sequence states. The proposed network encodes a set of robot trajectories which may contain states in common, and retrieves them accurately in the correct order. Further tests evaluate the fault-tolerance and noise sensitivity of the proposed model

Standard pattern recognition provides effective and noise-tolerant tools for machine learning tasks; however, most approaches only deal with real vectors of a finite and fixed dimensionality. In this tutorial paper, we give an overview about extensions of pattern recognition towards non-standard data which are not contained in a finite dimensional space, such as strings, sequences, trees, graphs, or functions. Two major directions can be distinguished in the neural networks literature: models can be based on a similarity measure adapted to non-standard data, including kernel methods for structures as a very prominent approach, but also alternative metric based algorithms and functional networks; alternatively, non-standard data can be processed recursively within supervised and unsupervised recurrent and recursive networks and fully recurrent systems.

In this tutorial paper about mathematical aspects of neural networks, we will focus on two directions: on the one hand, we will motivate standard mathematical questions and well studied theory of classical neural models used in machine learning. On the other hand, we collect some recent theoretical results (as of beginning of 2003) in the respective areas. Thereby, we follow the dichotomy offered by the overall network structure and restrict ourselves to feedforward networks, recurrent networks, and self-organizing neural systems, respectively.

Self-organization constitutes an important paradigm in machine learning with successful applications e.g. for data- and web-mining. However, so far most approaches have been proposed for data contained in a fixed and finite dimensional vector space. We will focus on extensions for more general data structures like sequences and tree structures in this article. Various extensions of the standard self-organizing map (SOM) to sequences or tree structures have been proposed in the literature: the temporal Kohonen map, the recursive SOM, and SOM for structured data (SOMSD), for example. These methods enhance the standard SOM by recursive connections. We define in this article a general recursive dynamic which enables the recursive processing of complex data structures based on recursively computed internal representations of the respective context. The above mechanisms of SOMs for structures are special cases of the proposed general dynamic, furthermore, the dynamic covers the supervised case of recurrent and recursive networks, too. The general framework offers a uniform notation for training mechanisms such as Hebbian learning and the transfer of alternatives such as vector quantization or the neural gas algorithm to structure processing networks. The formal definition of the recursive dynamic for structure processing unsupervised networks allows the transfer of theoretical issues from the SOM literature to the structure processing case. One can formulate general cost functions corresponding to vector quantization, neural gas, and a modification of SOM for the case of structures. The cost functions can be compared to Hebbian learning which can be interpreted as an approximation of a stochastic gradient descent. We derive as an alternative the exact gradien...

We review a recent extension of the self-organizing map (SOM) for temporal structures with a simple recurrent dynamics leading to sparse representations, which allows an efficient training and a combination with arbitrary lattice structures. We discuss its practical applicability and its theoretical properties. Afterwards, we put the approach into a general framework of recurrent unsupervised models. This generic formulation also covers a variety of well-known alternative approaches including the temporal Kohonen map, the recursive SOM, and SOM for structured data. Based on this formulation, mathematical properties of the models are investigated. Interestingly, the dynamic can be generalized from sequences to more general tree structures thus opening the way to unsupervised processing of general data structures.