by Fuchun Peng, Dale Schuurmans
In 6th Natural Language Processing Pacific Rim Symposium (NLPRS2001) Shai Fine, Yoram Singer, and Naftali Tishby.1998
http://ai2.math.uwaterloo.ca/~f3peng/publication/NLPRS01_Segmentation.ps
Add To MetaCart
Abstract:
We propose a simple two-level hierarchical probability model for unsupervised word segmentation. By treating words as strings composed of morphemes /phonemes which are themselves composed of character/phone strings, we use EM to rst identify the important morphemes/phonemes in a corpus, and then use a second level of EM to identify words given a lower level morpheme /phoneme segmentation. To further improve performance of the basic method we employ a mutual information criterion to eliminate long word agglomerations and reduce the size of the inferred lexicon while moving EM out of poor local maxima. Experiments on the Brown corpus show that our method accurately recovers hidden word boundaries using less training data than current MDL based approaches, even though our method is only trained on raw unsupervised data. 1
Citations
|
4345
|
Maximum likelihood from incomplete data via the EM algorithm
– Dempster, Laird, et al.
- 1977
|
|
58
|
An efficient, probabilistically sound algorithm for segmentation and word discovery
– Brent
- 1999
|
|
46
|
Structure learning in conditional probability models via an entropic prior and parameter extinction
– Brand
- 1999
|
|
42
|
Learning to segment speech using multiple cues: A connectionist model
– Christiansen, Allen, et al.
- 1998
|
|
30
|
The unsupervised acquisition of a lexicon from continuous speech
– Marcken
- 1995
|
|
27
|
Language modeling by variable length sequences: Theoretical formulation and evaluation of multigrams
– Deligne, Bimbot
- 1995
|
|
18
|
Self-supervised Chinese word segmentation
– Peng, Schuurmans
- 2001
|
|
14
|
Unsupervised learning of word boundary with description length gain
– Kit, Wilks
- 1999
|
|
8
|
Distributional regularity and phonotactics are useful for segmentation
– Brent, Cartwright
- 1996
|
|
8
|
Information extraction with HMMs and shrinkage
– Frietag, McCallum
- 1999
|
|
5
|
Coping with Variation in Speech Segmentation
– Christiansen, Allen
- 1997
|
|
4
|
Unsupervised word induction using MDL criterion
– Hua
- 2000
|
