Citation Context

...work might be integrated with distributional semantics models and techniques (Mitchell and Lapata 2010; Baroni and Lenci 2010; Erk, Padó, and Padó 2010; Clarke 2012; Boleda, im Walde, and Badia 2012; =-=Silberer and Lapata 2012-=-, inter alia). Finally, we note that, while our framework was applied to polysemous queries only, nothing prevents from using it to perform experiments at different levels of sense granularity. A qual...

Citation Context

...d we combined them in the most straightforward way. There is room for improvement by tuning the models to the task, or by using more sophisticated approaches to combine modality-specific information (=-=Silberer and Lapata, 2012-=-). We built our work on an existing corpus of highquality video material, which is restricted to the cooking domain. As a consequence, the corpus covers only a limited inventory of activity types and ...

Citation Context

...e of feature norms (McRae et al., 2005), which have been used in the cognitive science literature to approximate human perception and were recently used as a visual proxy in distributional semantics (=-=Silberer and Lapata, 2012-=-). We present the first effort, that we are aware of, for using feature norms to study image description generation. Such rich data allows us to develop significantly more comprehensive generation mod...

Citation Context

...: words in a document and a categorical label for the document. The same idea could be applied here: the latent semantic representation generates the observed brain activity and corpus statistics. Generative and discriminative models both have their own strengths and weaknesses, generative models being particularly strong when data sources are limited (Ng and Jordan, 2002). Our task is an interesting blend of data-limited and data-rich problem scenarios. In the past, various pieces of additional information have been incorporated into semantic models. For example, models with behavioral data (Silberer and Lapata, 2012) and models with visual information (Bruni et al., 2011; Silberer et al., 2013) have both shown to improve semantic representations. Other works have correlated VSMs built with text or images with brain activation data (Murphy et al., 2012b; Anderson et al., 2013). To our knowledge, this work is the first to integrate brain activation data into the construction of the VSM. 4 Data 4.1 Corpus Data The corpus statistics used here are the downloadable vectors from Fyshe et al. (2013)3. They are compiled from a 16 billion word subset of ClueWeb09 (Callan and Hoy, 2009) and contain two types of corp...

Citation Context

...notation of regions (Socher and Fei-Fei, 2010) and complete images (Hardoon et al., 2006; Hodosh et al., 2013). Given two paired observation matrices, in our case Vs and Ws, CCA aims at capturing the linear relationship that exists between these variables. This is achieved by finding a pair of matrices, in our 1407 case CV ∈ Rdv×d and CW ∈ Rdw×d, such that the correlation between the projections of the two multidimensional variables into a common, lower-rank space is maximized. The resulting multimodal space has been shown to provide a good approximation to human concept similarity judgments (Silberer and Lapata, 2012). In our setup, after applying CCA on the two spaces Vs and Ws, we obtain the two projection mappings onto the common space and thus our projection function can be derived as: fprojv→w = CV CW −1 (2) Singular Value Decomposition (SVD) SVD is the most widely used dimensionality reduction technique in distributional semantics (Turney and Pantel, 2010), and it has recently been exploited to combine visual and linguistic dimensions in the multimodal distributional semantic model of Bruni et al. (2014). SVD smoothing is also a way to infer values of unseen dimensions in partially incomplete matrice...

Citation Context

...cond, it has been found that meaning is often grounded in the perceptual system, and that the quality of semantic representations improves significantly when they are grounded in the visual modality (=-=Silberer and Lapata, 2012-=-; Bruni et al., 2014). Having an intermediate visual space means that words in different languages can be grounded in the same space. Third, it is natural to use vision as an intermediate: when we com...

Citation Context

...ons with extra-linguistic perceptual input. These models outperform language-only models on a range of tasks, including modelling semantic similarity and relatedness, and predicting compositionality (=-=Silberer and Lapata, 2012-=-; Roller and Schulte im Walde, 2013; Bruni et al., 2014). Although feature norms have also been used, raw image data has become the de-facto perceptual modality in multi-modal models. However, if the ...