Lawrence, S., Giles, C. L. & Fong, S. (2000), `Natural language grammatical inference with recurrent neural networks', IEEE Transactions on Knowledge and Data Engineering 12(1), 126--140.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....and learning rates in RTRL lead to a faster convergence at the cost of frequent signi cant error increases uctuations. On the other hand, smaller rates make training slow and susceptible to local minima. We tried to avoid these problems by resorting to annealed schedules for learning rate (e.g. [33]) Compared with the xed learning rate regime, a piece wise linear learning rate decrease: 0:05 0:001 over the rst 600.000 input symbol presentations and 0:001 0:0001 over the next 600.000 6.000.000 time steps, was found to give consistently improved RNN performance on the data sets ....

Steve Lawrence, C. Lee Giles, and Sandiway Fong, \Natural language grammatical inference with recurrent neural networks," IEEE Transactions on Knowledge and Data Engineering, vol. 12, no. 1, pp. 126-140, 2000.

....J. L. Garcia Rosa and E. Franozo The stone bought the man . 5) Barred metaphor, 5) is clearly anomalous and will cause HTRP to activate its error output. It has already been argued that learning grammar is impossible without negative examples, and the error output grants HTRP with such property [6]. As for architecture, the error output, which also has two hidden units, differs from the other networks at the input layer. It has 80 units (20 for the verb and 60 for nouns) instead of 40, since it is unknown which nouns, in conjunction with the verb, activate the error output. 2.3 How the ....

Lawrence, S., Giles, C. L., and Fong, S.: Natural Language Grammatical Inference with Recurrent Neural Networks. IEEE Transactions on Knowledge and Data Engineering, Vol. 12, No. 1 (2000) 126-140

....on the whole sentence in which the verb occurs. In sum, a nonlexicalist approach is preferable. 3 The Connectionist Architecture HTRP system uses a connectionist architecture representing eleven independent artificial neural networks, one for each thematic role and one for the error output [Lawrence et al. 1999]. The elementary processors are classical perceptron like units, and each net has 40 input units, 2 hidden units, and one output unit. The input units are responsible for the representation of two words of a sentence, the verb and one of the nouns. Since each HTRP sentence has, at most, three ....

....of these two microfeature sets (see figure 1) The error output, which has also two hidden units and one output unit, differs at the input layer, which in this case has 80 units, because it is unknown which nouns, in conjunction with the verb, activate the error output. 3. 1 The Error Output Lawrence et al. 1999] propose a recurrent neural network to classify English sentences as grammatical or un MACHINE LEARNING 854 grammatical, exhibiting the same discriminatory power supplied by linguistic theory. The network is not divided into innate and learned knowledge. Instead, positive and negative examples ....

S. Lawrence, C. L. Giles, and S. Fong. Natural Language Grammatical Inference with Recurrent Neural Networks. IEEE Trans. on Knowledge and Data Engineering (accepted for publication), 1999.

....cited the non generative nature of such connectionist models as a fundamental shortcoming of the entire field. Partly in response to these criticisms, many connectionists have spent the past decade investigating network models which support generativity through recurrent (feedback) connections (Lawrence, Giles, and Fong 1998; Rodriguez, Wiles, and Elman 1999; Williams and Zipser 1989) The research we present here is an attempt to contribute to this effort while focusing as strongly as possible on the natural language issues described above. Such an attempt faces a number of challenges. First, despite analysis of ....

Lawrence, S., C. Giles, and S. Fong (1998). Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, to appear.

.... this task are those from statistical language learning [2, 3, 5, 20] The basic connectionist approach for learning language is based around the SRN, in which the network is trained to predict the next word in a sentence [7, 8, 30] or else trained to assess whether a sentence is grammatical or not [24, 25]. However, the simple SRN has not produced results comparable with the statistical parsers, because its basic output representation is flat and unstructured. The reason the simple SRN does not produce structured output representations lies with the required number of relationships which must be ....

....in [16] have been achieved with a type A SSN. As noted in the Introduction to this article, experiments with natural language using SRNs have typically used a restricted form of input representation, either predicting the next word in a sentence [6, 8, 30] or assessing whether it is grammatical [24, 25]. Our extension to the SRN, the SSN, corrects this limitation by enhancing the range of output representations to include structured parse trees. Our approach is designed to generate a structured representation given a sequence of input data. The generation aspect of this task largely ....

S. Lawrence, C. L. Giles, and S. Fong. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, in press. 21

....same number of b s. The two tokens, a and b, were represented with [1 0] and [0 1] respectively. Each network was unique and had either di erent initial weights or was con gured with di erent learning parameters including learning rate ( xed at 0. 3 (FLR) or an adaptive strategy (ALR) described by Lawrence et al. 1998), number of activation copies saved for BPTT (ranging from 5 to 12) target codes (binary (BT) 1 0] 0 1] or soft (ST) 0.9 0.1] 0.1 0.9] These variations allowed us to study the impact of prior constraints. The logistic output function was used for all networks. No momentum was used. The ....

Lawrence, S., Giles, C. L. and Fong, S. (1998) Natural language grammatical inference with recurrent neural networks.

.... of recurrent neural networks (RNNs) Part of their appeal is the ability to incorporate syntax and semantics into a single encompassing model (Elman, 1991) They have also demonstrated competence in learning a wide range of grammatical structures, for example from an introductory linguistics text (Lawrence et al. 1998). Furthermore, they re ect human performance on a number of language tasks (Weckerly and Elman, 1992; Christiansen and Chater, 1998) and can account for historical descriptions of language change (Hare and Elman, 1995) As well as processing constraints, RNNs have learning constraints. That is, ....

Lawrence, S., Giles, C. L., and Fong, S. (1998). Natural language grammatical inference with recurrent neural networks. To appear: IEEE Transactions on Knowledge and Data Engineering.

....between training data and NN performance on NLP. Elman reported better performance when training with simple sentences before using more complex ones, 2] Laurence, Giles, and Fong, on the other hand, report that sectioning the training data in this manner decreased performance in their studies, [4]. The question then becomes how to choose a subset of the training data that will allow the NN to better learn the intended task, and how to present this given data. 2 The Task for the NN boy is the agent of runs . This task will require the NN to store time dependant information, since it ....

Lawrence, S., Giles, C., and Fong, S. Natural Language Grammatical Inference with Recurrent Neural Networks. Accepted for Publication, IEEE Transactions on Knowledge and Data Engineering. 1998.

....promise as computational models of various aspects of the human language processing system. Part of their major appeal is the ability to incorporate syntax and semantics into a single encompassing model [3] They have also demonstrated competence in learning a wide range of grammatical structures [7], and often reflect real world data on natural language tasks [2, 10] and language change [5] Although these results have been affected by it, the issue of constraints of RNNs has not been explicitly examined. It seems important then, to investigate the constraints of recurrent networks and the ....

S. Lawrence, C. L. Giles, and S. Fong. Natural language grammatical inference with recurrent neural networks. To appear: IEEE Transactions on Knowledge and Data Engineering, 1998.

....is then used 4 which is trained on the sequence of outputs from the SOM. The Elman network was chosen because it is suitable for the problem (a grammatical inference style problem) 13] and because it has been shown to perform well in comparison to other recurrent architectures (e.g. see [36]) The Elman neural network has feedback from each of the hidden nodes to all of the hidden nodes, as shown in figure 3. For the Elman network: O(k 1) C T z k c 0 (13) and z k = F n h (Az k 1 Bu k b) 14) where C is a n h n o vector representing the weights from the hidden layer to ....

Steve Lawrence, C. Lee Giles, and Sandiway Fong. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, 12(1):126--140, 2000.

.... language, without appealing to traditional linguistic concepts [5] 7] Despite the remarkable advances which have come out of connectionist research (e.g. 8] and the now common use of recurrent networks, and Simple Recurrent Networks (SRNs) 9] especially, in the study of language (e.g. [10]) recurrent neural networks suffer from particular problems which make them imperfectly suited to language tasks. The vast majority of work in this field employs small networks and datasets (usually artificial) and although many interesting linguistic issues may be thus tackled, real progress in ....

S. Lawrence, C. Lee Giles & S. Fong (in press). Natural language grammatical inference with recurrent neural networks. IEEE Trans. on knowledge and data engineering.

....(a 4 Elman refers to the topology of the network full backpropagation through time was used as opposed to the truncated version used by Elman. 9 grammatical inference style problem) 13] and because it has been shown to perform well in comparison to other recurrent architectures (e.g. see [36]) The Elman neural network has feedback from each of the hidden nodes to all of the hidden nodes, as shown in figure 3. For the Elman network: v H x r MhW (13) and MrH kz v 1M h x (14) where is a .1. vector representing the weights from the hidden layer to the output nodes, ....

Steve Lawrence, C. Lee Giles, and Sandiway Fong. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering. accepted for publication. 28

No context found.

Lawrence, S., Giles, C. L. & Fong, S. (2000), `Natural language grammatical inference with recurrent neural networks', IEEE Transactions on Knowledge and Data Engineering 12(1), 126--140.

No context found.

S. Lawrence, C. Giles, and S. Fong. Natural language grammatical inference with recurrent neural networks. In IEEE Transactions on knowledge and Data Engineering. IEEE Press, 1998. (accepted).

No context found.

S. Lawrence, C. L. Giles, and S. Fong. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, 12(1):126--140, 2000.

No context found.

Lawrence S., Giles L. and Fong S., Natural Language Grammatical Inference with Recurrent Neural Networks, IEEE Transactions on Knowledge and Data Engineering, 1998.

No context found.

Lawrence S., Giles L., Fong S., Natural Language Grammatical Inference with Recurrent Neural Networks, IEEE Transactions on Knowledge and Data Engineering, 1998.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC