Abstract

TextTiling is a technique for subdividing texts into multi-paragraph units that represent passages, or subtopics. The discourse cues for identifying major subtopic shifts are patterns of lexical co-occurrence and distribution. The algorithm is fully implemented and is shown to produce segmentation that corresponds well to human judgments of the subtopic boundaries of 12 texts. Multi-paragraph subtopic segmentation should be useful for many text analysis tasks, including information retrieval and summarization. 1.

Citations

1302	WordNet: an online lexical database - Miller, Beckwith, et al. - 1990
1095	Automatic text processing: The transformation, analysis, and retrieval of information by computer - Salton
527	Using information content to evaluate semantic similarity in a taxonomy - Resnik - 1995
500	Assessing agreement on classification tasks: the kappa statistic - Carletta - 1996
342	A trainable document summarizer - Kupiec, Pedersen, et al. - 1995
269	Lexical cohesion computed by thesaural relations as an indicator of the structure of text - Morris, Hirst - 1991
265	Word sense disambiguation using statistical models of Roget’s categories trained on large corpora - Yarowsky - 1992
260	Multi-Paragraph Segmentation of Expository Text - Hearst - 1994
238	TileBars: Visualization of Term Distribution Information in Full Text Information Access - Hearst - 1995
179	Passage-level evidence in document retrieval - Callan - 1994
172	One sense per discourse - Gale, Church, et al. - 1992
169	Subtopic Structuring for Full-length Document Access - Hearst, Plaunt - 1993
168	The first text retrieval conference - Harman - 1993
163	Constructing literature abstracts by computer: techniques and prospects - Paice - 1990
157	Discourse Markers - Schiffrin - 1987
154	Approaches to passage retrieval in full text information systems - Salton, Allan, et al. - 1993
118	How effective is suffixing - Harman - 1991
114	Mixed initiative in dialogue: An investigation into discourse segmentation - Walker, Whittaker - 1990
102	Empirical studies on the disambiguation of cue phrases - Hirschberg, Litman - 1993
92	Text segmentation based on similarity between words - Kozima - 1993
91	Estimating upper and lower bounds on the performance of word-sense disambiguation programs - Gale, Church, et al. - 1992
79	Notes on a schema for stories - Rumelhart - 1975
72	Automatic text decomposition using text segments and text themes - Salton, Singhal, et al. - 1996
52	Redundancy in collaborative dialogue - Walker - 1992
49	Document and passage retrieval based on hidden markov models - Mittendorf, Schäuble - 1994
46	Texttiling: A quantitative approach to discourse segmentation - Hearst - 1993
44	The Thread of Discourse - GRIMES - 1975
41	Retrieval of partial documents - Moffat, Sacks-Davis, et al. - 1993
41	Lexical cohesion, the thesaurus, and the structure of text - Morris, Hirst - 1991
40	An Automatic Method of Finding Topic Boundaries - Reynar - 1994
39	Context and structure in automated fulltext information access - Hearst - 1994
34	Informedia digital video library - Christel, Kanade, et al. - 1995
33	A compiler for two-level phonological rules - Karttunen, Koskenniemi, et al. - 1987
28	Instructions for annotating discourses - Nakatani, Grosz, et al. - 1995
27	Functional sentence perspective: A case study from Japanese and English. Linguistic Inquiry - Kuno - 1972
27	Selective text utilization and text traversal - Salton, Allan - 1995
27	Discourse deixis: Reference to discourse segments - Webber - 1988
25	The flow of thought and the flow of language - Chafe - 1979
23	The paragraph as a grammatical unit - Longacre - 1979
22	Tense trees as the “fine structure” of discourse - Hwang, Schubert - 1992
22	Aspects of Text Structure: An investigation of the lexical organization of text - Phillips - 1985
21	The comparison of percentages in matched samples - Cochran - 1950
21	The interpretation of tense in discourse - Webber - 1987
17	Towards New Measures of Information Retrieval Evaluation - Hersh, Molnar - 1995
17	Replicability of transaction and action coding in the map task corpus - Isard, Carletta - 1995
16	Stylistic variation in an information retrieval experiment - Karlgren - 1994
16	Grammatico-statistical approach to discourse partitioning - Nomoto, Nitta - 1994
15	Adaptive method of automatic abstracting and indexing - Skorochod’ko - 1972
14	Discourse Analysis. Cambridge Textbooks in Linguistics - Brown, Yule - 1998
11	The generation of high-level structure for extended explanations - Mooney, Carberry, et al. - 1990