DMCA
Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction
Cached
Download Links
| Venue: | Journal of Artificial Intelligence Research |
| Citations: | 3 - 1 self |
BibTeX
@ARTICLE{Eisenstein_modelinggesture,
author = {Jacob Eisenstein and Regina Barzilay and Randall Davis},
title = {Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction},
journal = {Journal of Artificial Intelligence Research},
year = {},
pages = {353--398}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and non-communicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature. 1 1.
Keyphrases
gesture salience coreference resolution keyframe extraction hidden variable little difficulty noun phrase conditional hidden-variable model irrelevant hand movement human intuition meaningful gesture semantic entity impede learning non-verbal modality natural language processing present conditional modality fusion gesture feature crucial information natural language coreference accord non-communicative hand motion human raters outperform summary significant gain automatic system multimedia keyframe summary coreference annotation
