DMCA
Action is in the Eye of the Beholder: Eye-gaze Driven Model for Spatio-Temporal Action Localization
Cached
Download Links
| Citations: | 4 - 1 self |
BibTeX
@MISC{Shapovalova_actionis,
author = {Nataliya Shapovalova and Michalis Raptis and Leonid Sigal and Greg Mori},
title = {Action is in the Eye of the Beholder: Eye-gaze Driven Model for Spatio-Temporal Action Localization},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We propose a weakly-supervised structured learning approach for recognition and spatio-temporal localization of actions in video. As part of the proposed approach, we develop a generalization of the Max-Path search algorithm which allows us to efficiently search over a structured space of multiple spatio-temporal paths while also incorporating context information into the model. Instead of using spatial annotations in the form of bounding boxes to guide the latent model during training, we utilize human gaze data in the form of a weak supervisory signal. This is achieved by incorporating eye gaze, along with the classification, into the structured loss within the latent SVM learning framework. Experiments on a challenging benchmark dataset, UCF-Sports, show that our model is more accurate, in terms of classification, and achieves state-of-the-art results in localization. In addition, our model can produce top-down saliency maps conditioned on the classification label and localized latent paths. 1
Keyphrases
spatio-temporal action localization eye-gaze driven model context information latent svm learning framework challenging benchmark dataset structured space weakly-supervised structured learning approach classification label human gaze data multiple spatio-temporal path max-path search algorithm eye gaze weak supervisory signal localized latent path spatio-temporal localization spatial annotation latent model state-of-the-art result top-down saliency map structured loss
