DMCA
Identifying Nuggets of Information in GALE Distillation Evaluation
BibTeX
@MISC{Babko-malaya_identifyingnuggets,
author = {Olga Babko-malaya and Greg Milette and Michael Schneider and Sarah Scogin},
title = {Identifying Nuggets of Information in GALE Distillation Evaluation},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
This paper describes an approach to automatic nuggetization and implemented system employed in GALE Distillation evaluation to measure the information content of text returned in response to an open-ended question. The system identifies nuggets, or atomic units of information, categorizes them according to their semantic type, and selects different types of nuggets depending on the type of the question. We further show how this approach addresses the main challenges for using automatic nuggetization for QA evaluation: the variability of relevant nuggets and their dependence on the question. Specifically, we propose a template-based approach to nuggetization, where different semantic categories of nuggets are extracted dependent on the template of a question. During evaluation, human annotators judge each snippet returned in response to a query as relevant or irrelevant, whereas automatic template-based nuggetization is further used to identify the semantic units of information that people would have selected as ‘relevant ’ or ‘irrelevant’ nuggets for a given query. Finally, the paper presents the performance results of the nuggetization system which compare the number of automatically generated nuggets and human nuggets and show that our automatic nuggetization is consistent with human judgments.
Keyphrases
gale distillation evaluation automatic nuggetization atomic unit semantic type information content human nugget human annotator main challenge nuggetization system selects different type semantic unit different semantic category qa evaluation human judgment relevant nugget template-based approach open-ended question irrelevant nugget implemented system whereas automatic template-based nuggetization performance result
