DMCA
Statistical evaluation of information distillation systems (2008)
| Venue: | Proceedings of LREC 2008 (Linguistic Resources and Evaluation Conference |
| Citations: | 4 - 0 self |
BibTeX
@INPROCEEDINGS{White08statisticalevaluation,
author = {J. V. White and D. Hunter and J. D. Goldstein},
title = {Statistical evaluation of information distillation systems},
booktitle = {Proceedings of LREC 2008 (Linguistic Resources and Evaluation Conference},
year = {2008}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We describe a methodology for evaluating the statistical performance of information distillation systems and apply it to a sim-ple illustrative example. (An information distiller provides written English responses to English queries based on automated searches/transcriptions/translations of English and foreign-language sources. The sources include written documents and sound tracks.) The evaluation methodology extracts information nuggets from the distiller response texts and gathers them into fuzzy equivalence classes called nugs. The methodology supports the usual performance metrics, such as recall and precision, as well as a new information-theoretic metric called proficiency, which measures how much information a distiller provides relative to all of the information provided by a col-lection of distillers working on a common query and corpora. Unlike previous evaluation techniques, the methodology evaluates the relevance, granularity, and redundancy of information nuggets explicitly. 1. Information Distillers An autonomous information distiller takeswritten queries as inputs, and in response, automatically gathers, transcribes, translates (if necessary), and distills relevant information frommultilingual text and speech sources. The distiller out-puts the distilled information in a readable document written
Keyphrases
information distillation system statistical evaluation information distiller information nugget sim-ple illustrative example speech source common query usual performance metric automated search transcription translation much information autonomous information distiller takeswritten query relevant information frommultilingual text english response readable document sound track previous evaluation technique foreign-language source distiller response text statistical performance evaluation methodology fuzzy equivalence class distilled information
