Abstract

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper introduces four different ROUGE measures: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S included in the ROUGE summarization evaluation package and their evaluations. Three of them have been used in the Document Understanding Conference (DUC) 2004, a large-scale summarization evaluation sponsored by NIST. 1

Citations

7321	Introduction to Algorithms - Cormen, Leiserson, et al. - 2001
275	Bootstrap methods and their application - Davison - 1997
260	and E.Hovy. Automatic Evaluation of Summaries Using N-gram Co-Occurrence Statistics - Lin - 2003
161	Automatic Summarization - Mani - 2001
65	Automatic Evaluation and Uniform Filter Cascades for Inducing N-best Translation Lexicons - Melamed - 1995
14	Meta-evaluation of Summaries in a Cross-lingual Environment using Content-based Metrics - Saggion, Radev, et al. - 2002
9	Evaluation of Text summarization in a Cross-lingual Information Retrieval Framework - Radev, Teufel, et al. - 2002
7	2003. Precision and recall of machine translation - Melamed, Green, et al.
7	An introduction to DUC-2003: intrinsic evaluation of generic news text summarization systems - Over, Yen - 2003
6	Looking for a few Good Metrics: ROUGE and its Evaluation - Lin
2	Och (2004) Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics - Lin, F