Sanjeev Khudanpur and Jun Wu. Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling. Computer Speech and Language, to appear.  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... corpus to obtain explicit values for all of the hidden features such as recovering syntactic structure by running a parser, or recovering semantic content by using a latent semantic indexer and then incorporating statistics over explicitly measured features as additional constraints in the model [2, 12, 13]. However, doing so explicitly is not always possible, and even if attempted, sparse data problems almost always immediately arise in such complex models. Consequently, the perplexity improvements or word error rate reductions obtained are often minimal. In this paper we address the question: is ....

....is in the form of a product over exponential functions of features. The key idea for ecient calculation is to push the sums in as far as possible when summing (marginalizing) out irrelevant terms. Since calculating feature expectations has the same computational cost as normalization [12], we only show how to do normalization eciently here. The normalization factor can be calculated eciently by sum product algorithm, that is, summing over all the links at each time slice and passing through the trellis nodes with the product of the weight to the ongoing nodes we obtain = w 2 ....

S. Khudanpur and J. Wu, \Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling," Computer Speech and Language, Vol. 14, No. 4, pp. 355-372, 2000

.... corpus to obtain explicit values for all of the hidden features such as recovering syntactic structure by running a parser, or recovering semantic content by using a latent semantic indexer and then incorporating statistics over explicitly measured features as additional constraints in the model [3, 14, 15]. However, doing so explicitly is not always possible, and even if attempted, sparse data problems almost always immediately arise in such complex models. Consequently, the perplexity improvements or word error rate reductions obtained are often minimal. In this paper we address the question: is ....

....is in the form of a product over exponential functions of features. The key idea for ecient calculation is to push the sums in as far as possible when summing (marginalizing) out irrelevant terms. Since calculating feature expectations has the same computational cost as normalization [14], we only show how to do normalization eciently here. The normalization factor can be calculated eciently by sum product algorithm, that is, summing over all the links at each time slice and passing through the trellis nodes with the product of the weight to the ongoing nodes we V 1 N 1 1 M ....

S. Khudanpur and J. Wu, (2000); Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling, Computer Speech and Language, Vol. 14, No. 4, pp. 355-372

.... deal with the sparse data problems in language modeling, such as various discounting methods, word clustering [BPd 92, Lee97] link grammars [LST92] sentence mixtures [IO99] decision trees, caching [JMRS91, 1 2 Fuchun Peng Cla99] skipping models [Ros94, SO00] maximum entropy models [Ros94, KW00] latent semantic analysis [Bel00] structured language models [CJ98, Cha01] neural network models [BDV01] and web data improved trig rams [ZR01] Word segmentation is another important task in natural language processing. Word segmentation has many applications such as continuous speech ....

....models have been proposed in the literature to improve the basic n gram models. These sophisticated techniques include link grammars [LST92] sentence mixtures [IO99] decision trees, clustering [BPd 92] caching [JMRS91, Cla99] skipping models [Ros94, SO00] maximum entropy models [Ros94, KW00] latent semantic analysis [Bel00] structured language models [CJ98, Cha01] neural network models [BDV01] and web data improved trig rams [ZR01] The two references [Ros00, Goo00] provide a thorough overview and systematic investigation of current techniques. Most of these methods attack the ....

[Article contains additional citation context not shown here]

S. Khudanpur and J. Wu. Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling. Computer Speech and Language, pages 355-372, 2000.

....for prediction, including local ones and globe ones. Maximum entropy models has been shown to be good at modeling these constraints in many NLP areas, such as name entity recognition [8] POS and word sense disambiguation [76] parsing [17, 76] attribute grammar [65] language modeling [49, 53, 80], text segmentation [3] word spelling check [72] machine translation [4] text classi cation [63] and information extraction [54] The standard ME method is a supervised approach, which means that we have to label the training data before we can use GIS or IIS to train the ME model. The ....

Khudanpur, S. and Wu, J. Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling. In Computer Speech and Language, pp. 355-372, Oct. 2000.

....the best way to think about estimating Pr 32 : 1. Global sentence information such as grammaticality or semantic coherence is awkward to encode in a conditional framework. Some grammatical structure was captured in the structured language model of [1] and in the conditional exponential model of [2]. But such structure had to be formulated in terms of partial parse trees and left to right parse states. Similarly, modeling of semantic coherence was attempted in the conditional exponential model of [3] but had to be restricted to a limited number of pairwise word correlations. 2. External ....

Sanjeev Khudanpur and Jun Wu. Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling. Computer Speech and Language, to appear.

....as constraints of the form fw1;w2 (w; h) 1 i w = w 2 and w 1 2 h. More explicit approaches to exploiting syntactic and semantic models use probabalistic parsers to uncover head words which can then be used as predictors[59] Using ME, these can be combined with conventional ngram constraints[60]. This work is especially interesting since it models longer range dependencies in a more principled way than triggers. In the longer term, the growing synergy between the statistical approaches to speech and computational linguistics should pay dividends in this area. 5 CONCLUSIONS This paper ....

S Khudanpur and J Wu. Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modelling. Computer, Speech and Language, 14(4):355-372, 2000.

....than the other two because its predicted vocabulary tends to be orders of magnitude larger. The parser is more vulnerable than the tagger because of a large number of possible conditioning events. Chelba and Jelinek [3] have used deleted interpolation for all three SLM modules. Wu and Khudanpur [4] report better results with maximum entropy estimation of only the predictor component of equation (8) We expect that improved smoothing of all the SLM modules will result in further improvement in LM performance. We conducted experiments on two different corpora the Penn treebank [5] portion ....

Khudanpur, S. and Wu, J., "Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language models," Computer Speech and Language, 14(4):355-372, 2000.

....than the other two because its predicted vocabulary tends to be orders of magnitude larger. The parser is more vulnerable than the tagger because of a large number of possible conditioning events. Chelba and Jelinek [3] have used deleted interpolation for all three SLM modules. Wu and Khudanpur [4] report better results with maximum entropy estimation of only the predictor component of equation (8) We expect that improved smoothing of all the SLM modules will result in further improvement in LM performance. We conducted experiments on two different corpora the Penn treebank [5] portion ....

Khudanpur, S. and Wu, J., "Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language models," Computer Speech and Language, 14(4):355-372, 2000.

....this study instead of building an interpolation model as in [1] During testing, each hypothesis is parsed by the parser and the ME syntactic model is invoked for each partial parse with its appropriate head words. The word probability is calculated by equations (1) 2) and (3) Please refer to [1, 8, 13] for details. 2.2 Integrating Topic Dependencies and Syntactic Dependencies by ME Method We build a language model P (w l jw l 2 ; w l 1 ; nt l 2 ; nt l 1 ; h l 2 ; h l 1 ; t) which is both topic dependent and syntactic structure dependent. To avoid the data sparseness problem, we seek a model ....

S. Khudanpur and J. Wu, Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling, To Appear in Computer Speech and Language.

No context found.

Sanjeev Khudanpur and Jun Wu. Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling. Computer Speech and Language, to appear.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC