Abstract

Detailed analysis of human action, such as action classi-fication, detection and localization has received increasing attention from the community; datasets like JHMDB have made it plausible to conduct studies analyzing the impact that such deeper information has on the greater action un-derstanding problem. However, detailed automatic segmen-tation of human action has comparatively been unexplored. In this paper, we take a step in that direction and pro-pose a hierarchical MRF model to bridge low-level video fragments with high-level human motion and appearance; novel higher-order potentials connect different levels of the supervoxel hierarchy to enforce the consistency of the hu-man segmentation by pulling from different segment-scales. Our single layer model significantly outperforms the cur-rent state-of-the-art on actionness, and our full model im-proves upon the single layer baselines in action segmenta-tion. 1.