An Empirical Study of Smoothing Techniques for Language Modeling

DMCA

An Empirical Study of Smoothing Techniques for Language Modeling (1998)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Stanley F. Chen

Citations:

1224 - 21 self

BibTeX

@TECHREPORT{Chen98anempirical,
    author = {Stanley F. Chen},
    title = {An Empirical Study of Smoothing Techniques for Language Modeling},
    institution = {},
    year = {1998}
}

OpenURL

Abstract

We present an extensive empirical comparison of several smoothing techniques in the domain of language modeling, including those described by Jelinek and Mercer (1980), Katz (1987), and Church and Gale (1991). We investigate for the first time how factors such as training data size, corpus (e.g., Brown versus Wall Street Journal), and n-gram order (bigram versus trigram) affect the relative performance of these methods, which we measure through the cross-entropy of test data. In addition, we introduce two novel smoothing techniques, one a variation of Jelinek-Mercer smoothing and one a very simple linear interpolation technique, both of which outperform existing methods. 1

Keyphrases

language modeling smoothing technique empirical study brown versus wall street journal test data training data size bigram versus trigram relative performance simple linear interpolation technique first time extensive empirical comparison n-gram order jelinek-mercer smoothing

DMCA