Citation Context

...ute similarities between each shot in the movie and each sentence in the book. For dialogs, we use several similarity measures each capturing a different level of semantic similarity. We compute BLEU =-=[23]-=- between each subtitle and book sentence to identify nearly identical matches. Similarly to [34], we use a tf-idf measure to find near duplicates but weighing down the influence of the less frequent w...

Citation Context

...etwork, inspired by the success of encoder-decoder models for neural machine translation [10, 2, 1, 31]. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) =-=[9]-=- and the gated recurrent unit (GRU) [3]. Both types of activation successfully solve the vanishing gradient problem, through the use of gates to control the flow of information. The LSTM unit explicit...

Citation Context

... from the sentence embedding model. For each query sentence on the left, we retrieve the 4 nearest neighbor sentences (by inner product) chosen from books the model has not seen before. the skip-gram =-=[22]-=- architecture for learning representations of words. In the word skip-gram model, a word wi is chosen and must predict its surrounding context (e.g. wi+1 and wi−1 for a context window of size 1). Our ...

Citation Context

...Most effort in the domain of vision and language has been devoted to the problem of image captioning. Older work made use of fixed visual representations and translated them into textual descriptions =-=[6, 16]-=-. Recently, several approaches based on RNNs emerged, generating captions via a learned joint image-text embedding [13, 11, 36, 21]. These approaches have also been extended to generate descriptions o...

Citation Context

...le captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions =-=[8, 26]-=-, and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure 1: Shot from the movie Gone Girl, along with the subtitle, aligned with the book. We reason abo...

Citation Context

...Most effort in the domain of vision and language has been devoted to the problem of image captioning. Older work made use of fixed visual representations and translated them into textual descriptions =-=[6, 16]-=-. Recently, several approaches based on RNNs emerged, generating captions via a learned joint image-text embedding [13, 11, 36, 21]. These approaches have also been extended to generate descriptions o...

Citation Context

...s not considered in this paper, which is explored in [14]. To construct an encoder, we use a recurrent neural network, inspired by the success of encoder-decoder models for neural machine translation =-=[10, 2, 1, 31]-=-. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) [9] and the gated recurrent unit (GRU) [3]. Both types of activation successfully solve the vanishing g...

Citation Context

...s not considered in this paper, which is explored in [14]. To construct an encoder, we use a recurrent neural network, inspired by the success of encoder-decoder models for neural machine translation =-=[10, 2, 1, 31]-=-. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) [9] and the gated recurrent unit (GRU) [3]. Both types of activation successfully solve the vanishing g...

Citation Context

...em which identified sets of visual and audio features to align movies and scripts without making use of the subtitles. Such alignment has been exploited to provide weak labels for person naming tasks =-=[5, 30, 25]-=-. Closest to our work is [34], which aligns plot synopses to shots in the TV series for story-based content retrieval. This work adopts a similarity function between sentences in plot synopses and sho...

Citation Context

...s not considered in this paper, which is explored in [14]. To construct an encoder, we use a recurrent neural network, inspired by the success of encoder-decoder models for neural machine translation =-=[10, 2, 1, 31]-=-. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) [9] and the gated recurrent unit (GRU) [3]. Both types of activation successfully solve the vanishing g...

Citation Context

...beddings with soft attention in order to align the words to image regions. Early work on movie-to-text alignment include dynamic time warping for aligning movies to scripts with the help of subtitles =-=[5, 4]-=-. Sankar et al. [28] further developed a system which identified sets of visual and audio features to align movies and scripts without making use of the subtitles. Such alignment has been exploited to...

Citation Context

...otten significant attention in the past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning =-=[13, 11, 36, 35, 21]-=-, alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure ...

Citation Context

...his dataset has 94 movies, and 54,000 described clips. We represent each movie clip as a vector corresponding to mean-pooled features across each frame in the clip. We used the GoogLeNet architecture =-=[32]-=- as well as hybrid-CNN [38] for extracting frame features. For DVS, we pre-processed the descriptions by removing names and replacing these with a someone token. The LSTM architecture in this work is ...

Citation Context

...and 54,000 described clips. We represent each movie clip as a vector corresponding to mean-pooled features across each frame in the clip. We used the GoogLeNet architecture [32] as well as hybrid-CNN =-=[38]-=- for extracting frame features. For DVS, we pre-processed the descriptions by removing names and replacing these with a someone token. The LSTM architecture in this work is implemented using the follo...

Citation Context

...important for applications such as social robotics or assistive driving. Combining images or videos with language has gotten significant attention in the past year, partly due to the creation of CoCo =-=[18]-=-, Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A [20, 19], visual model learning fro...

Citation Context

...s not considered in this paper, which is explored in [14]. To construct an encoder, we use a recurrent neural network, inspired by the success of encoder-decoder models for neural machine translation =-=[10, 2, 1, 31]-=-. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) [9] and the gated recurrent unit (GRU) [3]. Both types of activation successfully solve the vanishing g...

Citation Context

...otten significant attention in the past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning =-=[13, 11, 36, 35, 21]-=-, alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure ...

Citation Context

...ces conditioned on the representation of the encoder:∑ t logP (wti+1|w<ti+1,hi) + ∑ t logP (wti−1|w<ti−1,hi) (10) The total objective is the above summed over all such training tuples. Adam algorithm =-=[12]-=- is used for optimization. 4.2. Visual-semantic embeddings of clips and DVS The model above describes how to obtain a similarity score between two sentences, whose representations are learned from mil...

Citation Context

...otten significant attention in the past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning =-=[13, 11, 36, 35, 21]-=-, alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure ...

Citation Context

... to further speed up computation. Learning. Since ground-truth is only available for a sparse set of shots, we regard the states of unobserved nodes as hidden variables and learn the CRF weights with =-=[29]-=-. 5. Experimental Evaluation We evaluate our model on our dataset of 11 movie/book pairs. We train the parameters in our model (CNN and CRF) on Gone Girl, and test our performance on the remaining 10 ...

Citation Context

...otten significant attention in the past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning =-=[13, 11, 36, 35, 21]-=-, alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure ...

Citation Context

...ouns and pronouns in a caption and visual objects using several visual and textual potentials. Lin et al. [17] does so for videos. In [11], the authors use RNN embeddings to find the correspondences. =-=[37]-=- combines neural embeddings with soft attention in order to align the words to image regions. Early work on movie-to-text alignment include dynamic time warping for aligning movies to scripts with the...

Citation Context

...beddings with soft attention in order to align the words to image regions. Early work on movie-to-text alignment include dynamic time warping for aligning movies to scripts with the help of subtitles =-=[5, 4]-=-. Sankar et al. [28] further developed a system which identified sets of visual and audio features to align movies and scripts without making use of the subtitles. Such alignment has been exploited to...

Citation Context

...he past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment =-=[11, 15, 34]-=-, Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure 1: Shot from the movie G...

Citation Context

...der-decoder models for neural machine translation [10, 2, 1, 31]. Two kinds of activation functions have recently gained traction: long short-term memory (LSTM) [9] and the gated recurrent unit (GRU) =-=[3]-=-. Both types of activation successfully solve the vanishing gradient problem, through the use of gates to control the flow of information. The LSTM unit explicity employs a cell that acts as a carouse...

Citation Context

...s such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries =-=[17]-=-. ∗Denotes equal contribution Figure 1: Shot from the movie Gone Girl, along with the subtitle, aligned with the book. We reason about the visual and dialog (text) alignment between the movie and a bo...

Citation Context

...ly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A =-=[20, 19]-=-, visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure 1: Shot from the movie Gone Girl, alon...

Citation Context

...descriptions of short video clips [35]. In [24], the authors go beyond describing what is happening in an image and provide explanations about why something is happening. For text-to-image alignment, =-=[15, 7]-=- find correspondences between nouns and pronouns in a caption and visual objects using several visual and textual potentials. Lin et al. [17] does so for videos. In [11], the authors use RNN embedding...

Citation Context

...le captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A [20, 19], visual model learning from textual descriptions =-=[8, 26]-=-, and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure 1: Shot from the movie Gone Girl, along with the subtitle, aligned with the book. We reason abo...

Citation Context

...em which identified sets of visual and audio features to align movies and scripts without making use of the subtitles. Such alignment has been exploited to provide weak labels for person naming tasks =-=[5, 30, 25]-=-. Closest to our work is [34], which aligns plot synopses to shots in the TV series for story-based content retrieval. This work adopts a similarity function between sentences in plot synopses and sho...

Citation Context

...esence of characters in a scene to those in a chapter, as well as uses hand-crafted similarity measures between sentences in the subtitles and dialogs in the books, similarly to [34]. Rohrbach et al. =-=[27]-=- recently released the Movie Description dataset which contains clips from movies, each time-stamped with a sentence from DVS (Descriptive Video Service). The dataset contains clips from over a 100 mo...

Citation Context

...ention in order to align the words to image regions. Early work on movie-to-text alignment include dynamic time warping for aligning movies to scripts with the help of subtitles [5, 4]. Sankar et al. =-=[28]-=- further developed a system which identified sets of visual and audio features to align movies and scripts without making use of the subtitles. Such alignment has been exploited to provide weak labels...

Citation Context

...ly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment [11, 15, 34], Q&A =-=[20, 19]-=-, visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure 1: Shot from the movie Gone Girl, alon...

Citation Context

...d might vary in the storyline from their movie release. Furthermore, we use learned neural embeddings to compute the similarities rather than hand-designed similarity functions. Parallel to our work, =-=[33]-=- aims to align scenes in movies to chapters in the book. However, their approach operates on a very coarse level (chapters), while ours does so on the sentence/paragraph level. Their dataset thus eval...

Citation Context

...s based on RNNs emerged, generating captions via a learned joint image-text embedding [13, 11, 36, 21]. These approaches have also been extended to generate descriptions of short video clips [35]. In =-=[24]-=-, the authors go beyond describing what is happening in an image and provide explanations about why something is happening. For text-to-image alignment, [15, 7] find correspondences between nouns and ...

Citation Context

...he past year, partly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The field has tackled a diverse set of tasks such as captioning [13, 11, 36, 35, 21], alignment =-=[11, 15, 34]-=-, Q&A [20, 19], visual model learning from textual descriptions [8, 26], and semantic visual search with natural multisentence queries [17]. ∗Denotes equal contribution Figure 1: Shot from the movie G...