Abstract

Images and videos are often characterized by multiple types of local descriptors such as SIFT, HOG and HOF, each of which describes certain aspects of object feature. Recognition systems benefit from fusing multiple types of these descriptors. Two widely applied fusion pipelines are descriptor concatenation and kernel average. The first one is effective when different descriptors are strongly corre-lated, while the second one is probably better when de-scriptors are relatively independent. In practice, however, different descriptors are neither fully independent nor fully correlated, and previous fusion methods may not be satis-fying. In this paper, we propose a new global representa-tion, Multi-View Super Vector (MVSV), which is composed of relatively independent components derived from a pair of descriptors. Kernel average is then applied on these com-ponents to produce recognition result. To obtain MVSV, we develop a generative mixture model of probabilistic canoni-cal correlation analyzers (M-PCCA), and utilize the hidden factors and gradient vectors of M-PCCA to construct MVSV for video representation. Experiments on video based ac-tion recognition tasks show that MVSV achieves promising results, and outperforms FV and VLAD with descriptor con-catenation or kernel average fusion strategy. 1.