Citation Context

...ers and a fully-connected layer at the end, each of which is followed by a non-linearity. This encoding can be viewed as early-fusion [14] (other types of fusions, e.g., late-fusion or 3D convolution =-=[35]-=- can also be applied to this architecture). Recurrent encoding takes one frame as an input for each time-step and extracts spatio-temporal features using an RNN in which the temporal dynamics is model...

Citation Context

... description datasets, Montreal Video Annotation Dataset (M-VAD) [27] and MPII Movie Description Corpus (MPII-MD) [20], are included for evaluation on sentence generation. The two datasets are both composed of Hollywood movie snippets with descriptions from movie scripts or audio descriptions. Specifically, M-VAD contains about 49,000 video clips from 92 movies and MPII-MD contains about 68,000 video clips from 94 movies. We compare our LSTM-E approach with two 2-D CNN of AlexNet [14] and the 19-layer VGG [23] network both pre-trained on Imagenet ILSVRC12 dataset [22], and one 3- D CNN of C3D [28] pre-trained on Sports-1M video dataset [11]. Specifically, we take the output of 4096-way fc7 layer from AlexNet, 4096-way fc6 layer from the 19-layer VGG, and 4096-way fc6 layer from C3D as the frame/clip representation, respectively. The dimensionality of the visualsemantic embedding space and the size of hidden layer in LSTM are both set to 512. The tradeoff parameter λ leveraging the relevance loss and coherence loss is empirically set to 0.7. The sensitivity of λ will be discussed later. 5.2. Performance Comparison We empirically verify the merit of our LSTM-E model from two aspects: SVO...

Citation Context

...time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end-to-end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art. 1 Introduction Action recognition in video is an intensively researched area, with many recent approaches focused on application of Convolutional Networks (ConvNets) to this task, e.g. [13, 20, 26]. As actions can be understood as spatiotemporal objects, researchers have investigated carrying spatial recognition principles over to the temporal domain by learning local spatiotemporal filters [13, 25, 26]. However, since the temporal domain arguably is fundamentally different from the spatial one, different treatment of these dimensions has been considered, e.g. by incorporating optical flow networks [20], or modelling temporal sequences in recurrent architectures [4, 18, 19]. Since the introduction of the “AlexNet” architecture [14] in the 2012 ImageNet competition, ConvNets have dominat...

Citation Context

...hoice of the ConvNet, we employ Googlenet-bu4k, a variant of Googlenet [10] trained by Mettes et al. [7] using a bottom-up reorganization of the ImageNet hierarchy. Our experiments confirm that this model, though originally aiming for video event recognition, is better than its standard counterpart for video captioning as well. The pool5 layer after ReLU is used, resulting in a feature vector of 1,024 dimensions. Observing the increasing popularity of 3-D ConvNets (C3D) for video captioning [9, 15], we have also experimented with a C3D model trained by Tran et al. on one million sports videos [11]. Though being longer (4,096-dim), the C3D feature is inferior to the Googlenetbu4k feature according to our experiments. Moreover, in 1082 ConvNet Video Tagging fashion walking model ... Mean Pooling LSTM LSTM LSTM START models are LSTM Word embedding models are walking LSTM END... ... runway Word2VisualVec ... * models are walking down the runway * models are walking on the runway * a woman is walking down a runway * a woman is dancing * models are walking in a fashion show * models are walking on the ramp models are walking in a fashion show Video frames V isu a l e m b e d d in g Early Emb...

Citation Context

...saturation to 32 levels. For DSIFT features, we use Fisher Vector encoding to construct the video-level features. Then kernel PCA is utilized to reduce the dimensionality into 4096. DCNN Features: To explore the performance of different pre-trained CNN models, we extract multiple layers from different CNN models including inception-v3 [9], VGG16 and VGG-19 [10]. All the CNN features are applied with mean pooling to generate video-level representations. 2.3 Motion Modality To exploit the temporal information in the video, we extract the improved Dense Trajectory (iDT) [11] and the C3D features [12].For iDT features, HOG, HOF and MBH features are densely extracted from the video and encoded with Fisher Vector. Then kernel PCA is used to reduce dimensionality into 4096. For C3D features, we extract activations from the penultimate layer for every non-overlap 16 frames and use mean pooling to aggregate them into one vector. The challenge also provides baseline features [13] for global subtask, which consists of acoustic and visual features. 3. EXPERIMENTS 3.1 Experimental Setting In the global subtask, there are 9,800 video clips from 160 movies in the development set. We randomly select 6...