Download:
by Mikael Bodén, Alan Blair
Applied Intelligence
http://www.itee.uq.edu.au/~mikael/papers/spiral4.pdf
Add To MetaCart
Abstract:
Recent work by Siegelmann has shown that the computational power of recurrent neural networks matches that of Turing Machines. One important implication is that complex language classes (infinite languages with embedded clauses) can be represented in neural networks. Proofs are based on a fractal encoding of states to simulate the memory and operations of stacks. In the present work, it is shown that similar stack-like dynamics can be learned in recurrent neural networks from simple sequence prediction tasks. Two main types of network solutions are found and described qualitatively as dynamical systems: damped oscillation and entangled spiraling around fixed points. The potential and limitations of each solution type are established in terms of generalization on two different context-free languages. Both solution types constitute novel stack implementations – generally in line with Siegelmann’s theoretical work – which supply insights into how embedded structures of languages can be handled in analog hardware.
Citations
|
450
|
Fractals Everywhere
– Barnsley
- 1988
|
|
322
|
A learning algorithm for continually running fully recurrent neural networks
– Williams, Zipser
- 1990
|
|
247
|
Syntactic structures
– Chomsky
- 1957
|
|
221
|
Learning and development in neural networks: The importance of starting small
– Elman
- 1993
|
|
193
|
The induction of dynamical recognizers
– Pollack
- 1991
|
|
164
|
An Introduction to Chaotic Dynamical Systems
– Devaney
|
|
152
|
On the computational power of neural nets
– Siegelmann, Sontag
- 1995
|
|
102
|
The dynamics of discrete-time computation, with application to recurrent neural networks and �nite state machine extraction
– Casey
- 1996
|
|
94
|
Neural Networks and Analog Computation: Beyond the Turing Limit. Birkhäuser
– Siegelmann
- 1999
|
|
72
|
Toward a connectionist model of recursion in human linguistic performance
– Christiansen, Chater
- 1999
|
|
51
|
A recurrent neural network that learns to count
– RODRIGUEZ, WILES, et al.
- 1999
|
|
45
|
Dynamical recognizers: Real-time language recognition by analog computers
– Moore
- 1998
|
|
34
|
Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent networks
– Wiles, Elman
- 1995
|
|
31
|
Finite state machines and recurrent neural networks -- automata and dynamical systems approaches
– Tino, Horne, et al.
- 1998
|
|
25
|
Context-free and context-sensitive dynamics in recurrent neural networks
– Boden, Wiles
- 2000
|
|
25
|
Designing a counter: Another case study of dynamics and activation landscapes in recurrent networks
– Holldobler, Kalinke, et al.
- 1997
|
|
19
|
A recurrent network that performs a contextsensitive prediction task
– Steijvers, Grunwald
- 1996
|
|
18
|
Recurrent neural networks can learn to implement symbol-sensitive counting
– Rodriguez, Wiles
- 1998
|
|
17
|
Learning to predict a context-free language: Analysis of dynamics in recurrent hidden units
– Bodén, Wiles, et al.
- 1999
|
|
17
|
Inductive bias in context-free language learning
– Tonkes, Blair, et al.
- 1998
|
