Distinctions have been proposed between systems of reasoning for centuries. This article distills properties shared by many of these distinctions and characterizes the resulting systems in light of recent findings and theoretical developments. One system is associative because its computations reflect similarity structure and relations of temporal contiguity. The other is "rule based " because it operates on symbolic structures that have logical content and variables and because its computations have the properties that are normally assigned to rules. The systems serve complementary functions and can simultaneously generate different solutions to a reasoning problem. The rule-based system can suppress the associative system but not completely inhibit it. The article reviews evidence in favor of the distinction and its characterization. One of the oldest conundrums in psychology is whether people are best conceived as parallel processors of information who operate along diffuse associative links or as analysts who operate by deliberate and sequential manipulation of internal representations. Are inferences drawn through a network of learned associative pathways or through application of a kind of "psychologic"

In algorithmic music composition, a simple technique involves selecting notes sequentially according to a transition table that specifies the probability of the next note as a function of the previous context. I describe an extension of this transition table approach using a recurrent autopredictive connectionist network called CONCERT. CONCERT is trained on a set of pieces with the aim of extracting stylistic regularities. CONCERT can then be used to compose new pieces. A central ingredient of CONCERT is the incorporation of psychologically-grounded representations of pitch, duration, and harmonic structure. CONCERT was tested on sets of examples artificially generated according to simple rules and was shown to learn the underlying structure, even where other approaches failed. In larger experiments, CONCERT was trained on sets of J. S. Bach pieces and traditional European folk melodies and was then allowed to compose novel melodies. Although the compositions are occasionally pleasa...

In order for neural networks to learn complex languages or grammars, they must have sufficient computational power or resources to recognize or generate such languages. Though many approaches to effectively utilizing the computational power of neural networks have been discussed, an obvious one is to couple a recurrent neural network with an external stack memory- in effect creating a neural network pushdown automata (NNPDA). This NNPDA generalizes the concept of a recurrent network so that the network becomes a more complex computing structure. This paper discusses in detail a NNPDA- its construction, how it can be trained and how useful symbolic information can be extracted from the trained network. To effectively couple the external stack to the neural network, an optimization method is developed which uses an error function that connects the learning of the state automaton of the neural network to the learning of the operation of the external stack: push, pop, and no-operation. To minimize the error function using gradient descent learning, an analog stack is designed such that the action and storage of information in the stack are continuous. One interpretation of a continuous stack is the probabilistic storage of and action on data. After training on sample strings of an unknown source grammar, a quantization procedure extracts from the analog stack and neural network a discrete pushdown automata (PDA). Simulations show that in learning deterministic context-free grammars- the balanced parenthesis language, 1 n 0 n, and the deterministic Palindrome- the extracted PDA is correct in the sense that it can correctly recognize unseen strings of arbitrary length. In addition, the extracted PDAs can be shown to be identical or equivalent to the PDAs of the source grammars which were used to generate the training strings.

Artificial neural networks (ANN's), due to their inherent parallelism, offer an attractive paradigm for implementation of symbol processing systems for applications in computer science and artificial intelligence. This paper explores systematic synthesis of modular neural-network architectures for syntax analysis using a prespecified grammar---a prototypical symbol processing task which finds applications in programming language interpretation, syntax analysis of symbolic expressions, and high-performance compilers. The proposed architecture is assembled from ANN components for lexical analysis, stack, parsing and parse tree construction. Each of these modules takes advantage of parallel content-based pattern matching using a neural associative memory. The proposed neural-network architecture for syntax analysis provides a relatively efficient and high performance alternative to current computer systems for applications that involve parsing of LR grammars which constitute a widely used subset of deterministic context-free grammars. Comparison of quantitatively estimated performance of such a system [implemented using current CMOS very large scale integration (VLSI) technology] with that of conventional computers demonstrates the benefits of massively parallel neuralnetwork architectures for symbol processing applications.

Current connectionist parsing systems lack the ability to parse sentences of arbitrary length and to compute complex syntax trees. We propose that by using symbolic procedures within an connectionist architecture these problems can be solved. In addition, symbolic procedures can be used to hardwire a priori knowledge about the problem domain into the system. By doing this we get smaller networks which are easy to train. In the ProPars system symbolic procedures are used to implement a hybrid architecture that can parse sentences of arbitrary length, compute complex syntax trees, and integrate semantic and prosodic information from the speech signal into the parsing process. 1 Introduction Linguistic theory is traditionally divided into subfields such as phonetics, phonology, morphology, syntax, semantics, and pragmatics. Though each of these fields have independently developed quite powerful theories, interactions between different fields or the integration into one theory are still n...

: Connectionist network learning of context free languages has so far been applied only to very simple cases and has often made use of an external stack. Learning complex context free languages with a homogeneous neural mechanism looks like a much harder problem. The current paper takes a step toward solving this problem by analyzing context free grammar computation (without addressing learning) in a class of analog computers called Dynamical Automata, which are naturally implemented in connectionist networks. The result is a widely applicable method of using fractal sets to organize infinite state computations in a bounded state space. An appealing consquence is the development of parameter-space maps, which locate various complex computers in spatial relationships to one another. An example suggests that such a global perspective on the organization of the parameter space may be helpful for solving the hard problem of getting connectionist networks to learn complex grammars from exam...

Researchers often try to understand the representations that develop in the hidden layers of a neural network during training. Interpretation is difficult because the representations are typically highly distributed and continuous. By "continuous," we mean that if one constructed a scatter plot over the hidden unit activity space of patterns obtained in response to various inputs, examination at any scale would reveal the patterns to be broadly distributed over the space. Such continuous representations are naturally obtained if the input space and activation dynamics are continuous. Continuous representations are not always appropriate. Many task domains might benefit from discrete representations -- representations selected from a finite set of alternatives. Example domains include finite-state machine emulation, data compression, language and higher cognition (involving discrete symbol processing), and categorization. In such domains, standard neural...

In this paper we outline hybrid approaches to arti cial neural network-based natural language processing. We start by motivating hybrid symbolic/connectionist processing. Then we suggest various types of symbolic/connectionist integration for language processing: connectionist structure architectures, hybrid transfer architectures, hybrid processing architectures. Furthermore, we focus particularly on loosely coupled, tightly coupled, and fully integrated hybrid processing architectures. We give particular examples of these hybrid processing architectures and argue that the hybrid approach to arti cial neural network-based language processing has a lot of potential to overcome the gap between a neural level and a symbolic conceptual level. ii 1 Motivation for hybrid symbolic/connectionist processing In recent years, the eld of hybrid symbolic/connectionist processing has seen a remarkable

Neural network learning of context free languages has been applied only to very simple languages and has often made use of an external stack. Learning complex context free languages with a homogeneous neural mechanism looks like a much harder problem. The current paper takes a step toward solving this problem by analyzing context free grammar computation (without addressing learning) in a class of analog computers called Dynamical Automata, which are naturally implemented in neural networks. The result is a widely applicable method of using fractal sets to organize infinite state computations in a bounded state space. This method leads to a map of the locations of various context free grammars in the parameter space of one dynamical automaton/neural net. The map provides a global view of the parameterization problem which complements the local view of gradient descent methods. 1. Introduction A number of researchers have studied the induction of context free grammars by neural network...

A concept of accepting colonies is introduced. A hybrid connectionistsymbolic architecture ("neural pushdown automaton") for inference of colonies based on presentation of positive and negative examples of strings is then described, together with an algorithm for extracting a colony from trained neural network. Some examples of the inference of colonies generating /accepting simple context-free languages illustrate the function of the architecture. Keywords: grammar system, artificial neural network, pushdown automaton, colony, grammatical inference. 1 Introduction The problem of grammatical inference is generally hard and even for regular languages it is NP in the worst cases. There have been various heuristic methods developed, trying to find a suitable solution with reasonable computational expenses. We will focus our attention on hybrid architectures coupling principles of neural and symbolic computation. There have been many such architectures presented, concerning mostly (but no...