Abstract

Words unknown to the lexicon present a substantial problem to NLP modules that rely on mor-phosyntactic information, such as part-of-speech taggers or syntactic parsers. In this paper we present a technique for fully automatic acquisition of rules that guess possible part-of-speech tags for unknown words using their starting and ending segments. The learning is performed from a general-purpose lexicon and word frequencies collected from a raw corpus. Three complimentary sets of word-guessing rules are statistically induced: prefix morphological rules, suffix morpho-logical rules and ending-guessing rules. Using the proposed technique, unknown-word-guessing rule sets were induced and integrated into a stochastic tagger and a rule-based tagger, which were then applied to texts with unknown words. 1.

Citations

1654	B.: Building a large annotated corpus of english: The Penn treebank - Marcus, Marcinkiewicz, et al. - 1993
662	Transformation-Based Error-Driven Learning and Natural Language Processing A Case Study - Brill - 1995
433	A simple rule-based part of speech tagger - Brill - 1992
325	A Practical Part-of-Speech Tagger - Cutting, Kupiec, et al. - 1991
259	Frequency analysis of English usage: Lexicon and grammar - Francis, Kucera - 1982
227	Some advances in transformation-based part of speech tagging - Brill
177	Robust part-of-speech tagging using hidden Markov model - Kupiec - 1992
148	Grammatical category disambiguation by statistical optimization - DeRose - 1988
126	Coping with Ambiguity and Unknown Words through Probabilistic Models - Weischedel, Meteer, et al. - 1993
104	Enriching Linguistics with Statistics: Performance models of Natural Language. PhD dissertation. ILLC dissertation series 1995-14 - Bod - 1995
48	Automatic Stochastic Tagging of Natural Language - Dermatas, Kokkinakis - 1995
36	CELEX: A guide for users - Burnage - 1990
17	Estimating lexical priors for low-frequency morphologically ambiguous forms - Baayen, Sproat - 1996
15	Unsupervised Learning of Word-Category Guessing Rules - Mikheev - 1996
13	Combining linguistic knowledge and statistical learning in French part-of-speech tagging - Tzoukermann, Radev, et al. - 1995
12	Part of Speech Tagging with Neural Networks - Schmid - 1994
10	A SyntaxBased Part-of-Speech Analyser - Voutilainen - 1995
5	Learning part-of-speech guessing rules from lexicon: Extension to non-concatenative operations - Mikheev - 1996
2	A stochastic parts program and noun-phrase parser for unrestricted text - Church - 1988
1	Frequency Analysis of English Usage Lexicon and Grammar - Hayslett - 1981
1	Evaluating text categorization. Speech and Natural Language: Proceedings of a Workshop held at Pacific Grove CA - Lewis - 1991
1	An Introduction to the Bootstrap - Francis, Kucera - 1993
1	Frequency Analysis of English Usage:Lexicon and Grammar - Hayslett - 1981