We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focussing on the case of non linear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker'sinfomax principle) leads to a factorial code - hence to the same solution as required by the redundancy reduction principle of Barlow. We show also that this result is valid for linear, more generally unbounded, transfer functions, provided optimization is performed under an additive constraint, that is which can be written as a sum of terms, each one being specific to one output neuron. Finally we study the effect of a non zero input noise. We find that, at first order in the input noise, assumed to be small ...

A general-purpose object indexing technique is described that combines the virtues of principal component analysis with the favorable matching properties of high-dimensional spaces to achieve high precision recognition. An object is represented by a set of high-dimensional iconic feature vectors comprised of the responses of derivative of Gaussian filters at a range of orientations and scales. Since these filters can be shown to form the eigenvectors of arbitrary images containing both natural and man-made structures, they are well-suited for indexing in disparate domains. The indexing algorithm uses an active vision system in conjunction with a modified form of Kanerva’s sparse distributed memory which facilitates interpolation between views and provides a convenient platform for learning the association between an object’s appearance and its identity. The robustness of the indexing method was experimentally confirmed by subjecting the method to a range of viewing conditions and the accuracy was verified using a well-known model database containing a number of complex 3D objects under varying pose. 1

If different causes can interact on any occasion to generate a set of patterns, then systems modelling the generation have to model the interaction too. We discuss a way of combining multiple causes that is based on the Integrated Segmentation and Recognition architecture of Keeler, Rumelhart and Leow (1991). It is more co-operative than the scheme embodied in the mixture of experts architecture, which insists that just one cause generate each output, and more competitive than the noisy-or combination function which was recently suggested by Saund (1994a;b). Simulations confirm its efficacy. 1

The CA3 network in the hippocampus may operate as an autoassociator, in which declarative memories, known to be dependent on hippocampal processing, could be stored, and subsequently retrieved, using modifiable synaptic efficacies in the CA3 recurrent collateral system. On the basis of this hypothesis, the authors explore the computational relevance of the extrinsic afferents. to the CA3 network. A quantitative statistical analysis of the information that may be relayed by such afferent connections reveals the need for two distinct systems of input synapses. The synapses of the first system need to be strong (but not associatively modifiable) in order to force, during learning, the CA3 cells into a pattern of activity relatively independent of any inputs being received from the recurrent collaterals, and which thus reflects sizable amounts of new information. It is proposed that the mossy fiber system performs this function. A second system, with a large number of associatively modifiable synapses on each receiving cell, is needed in order to relay a signal specific enough to initiate the retrieval process. This may be identified, we propose, with the perforant path input to CA3. Key words: hippocampus, autoassociative memory, attractor neural networks, associative synapses, information storage

Abstract 1 Motivated in part by the hierarchical organization of the neocortex, a number of recently proposed algorithms have tried to learn hierarchical, or “deep, ” structure from unlabeled data. While several authors have formally or informally compared their algorithms to computations performed in visual area V1 (and the cochlea), little attempt has been made thus far to evaluate these algorithms in terms of their fidelity for mimicking computations at deeper levels in the cortical hierarchy. This thesis describes an unsupervised learning model that faithfully mimics certain properties of visual area V2. Specifically, we develop a sparse variant of the deep belief networks described by Hinton et al. (2006). We learn two layers of representation in the network, and demonstrate that the first layer, similar to prior work on sparse coding and ICA, results in localized, oriented, edge filters, similar to the gabor functions known to model simple cell receptive fields in area V1. Further, the second layer in our model encodes various combinations of the first layer responses in the data. Specifically, it picks up both collinear (“contour”) features as well as corners and junctions. More interestingly, in a quantitative comparison, the encoding of these more complex “corner ” features matches well with the results from Ito & Komatsu’s study of neural responses to angular stimuli in area V2 of the macaque. This suggests that our sparse variant of deep belief networks holds promise for modeling more higher-order features that are encoded in visual cortex. Conversely, one may also interpret the results reported here as suggestive that visual area V2 is performing computations on its input similar to those performed in (sparse) deep belief networks. This plausible relationship generates some intriguing hypotheses about V2 computations. 1 This thesis is an extended version of an earlier paper by Honglak Lee, Chaitanya Ekanadham, and Andrew Ng titled “Sparse deep belief net model for visual area V2.” 1

. We address the problem of computing a general-purpose early visual representation that satisfies two criteria. 1) Explicitness: To be more useful than the original pixel array, the representation must take a significant step toward making important image structure explicit. 2) Completeness: To support a diverse set of high-level tasks, the representation must not discard information of potential perceptual relevance. The most prevalent representation in image processing and computer vision that satisfies the completeness criterion is the wavelet code. In this paper, we propose a very different code which represents the location of each edge and the magnitude and blur scale of the underlying intensity change. By making edge structure explicit, we argue that this representation better satisfies the first criterion than do wavelet codes. To address the second criterion, we study the question of how much visual information is lost in the representation. We report a novel method for inver...

Knill et al.

Spatiotemporal connectionist networks (STCN's) comprise an important class of neural models that can deal with patterns distributed both in time and space. In this paper, we widen the application domain of the taxonomy for supervised STCN's recently proposed by Kremer (2001) to the unsupervised case. This is possible through a reinterpretation of the state vector as a vector of latent (hidden) variables, as proposed by Meinicke (2000). The goal of this generalized taxonomy is then to provide a nonlinear generative framework for describing unsupervised spatiotemporal networks, making it easier to compare and contrast their representational and operational characteristics. Computational properties, representational issues and learning are also discussed and a number of references to the relevant source publications are provided. It is argued that the proposed approach is simple and more powerful than the previous attempts, from a descriptive and predictive viewpoint. We also discuss the relation of this taxonomy with automata theory and state space modeling, and suggest directions for further work.

Abstract not found

We consider a linear, one-layer feedforward neural network performing a coding task. The goal of the network is to provide a statistical neural representation that convey as much information as possible on the input stimuli in noisy conditions. We determine the family of synaptic couplings that maximizes the mutual information between input and output distribution. Optimization is performed under different constraints on the synaptic efficacies. We analyze the dependence of the solutions on input and output noises. This work goes beyond previous studies of the same problem in that: (i) we perform a detailed stability analysis in order to find the global maxima of the mutual information; (ii) we examine the properties of the optimal synaptic configurations under different constraints; (iii) we do not assume translational invariance of the input data, as it is usually done when input are assumed to be visual stimuli. 1 Introduction This paper deals with the problem of learning the sta...