Citation Context

...proposed to tailor different task-specific requirements, including MatrixVector RNN (Socher et al., 2012) that represents every word as both a vector and a matrix, or Recursive Neural Tensor Network (=-=Socher et al., 2013-=-) that allows the model to have greater interactions between the input vectors. Many tasks have benefited from the recursive framework, including parsing (Socher et al., 2011b), sentiment analysis (So...

Citation Context

...e understanding and machine learning. In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks =-=[1]-=-, recurrent networks [2], convolutional networks [3, 4] and recursive-convolutional methods [5, 6] among others. All of these methods produce sentence representations that are passed to a supervised t...

Citation Context

...njoyed success in several NLP tasks [1, 2]. More recently, the use of distributed representations have been extended to model concepts beyond the word level, such as sentences, phrases and paragraphs =-=[3, 4, 5, 6]-=-, entities and relationships [7, 8] and embeddings of semantic categories [9, 10]. In this paper we propose a general framework for learning distributed representations of attributes: characteristics ...

Citation Context

...syntactic and semantic information of the phrasal translation rules. Therefore, in order to successfully apply DNN to model the whole translation process, such as modelling the decoding process, learning compact vector representations for the basic phrasal translation units is the essential and fundamental work. In this paper, we explore the phrase embedding, which represents a phrase (sequence of words) with a real-valued vector. In some previous works, phrase embedding has been discussed from different views. Socher et al. (2011) make the phrase embeddings capture the sentiment information. Socher et al. (2013a) enable the phrase embeddings to mainly capture the syntactic knowledge. Li et al. (2013) attempt to encode the reordering pattern in the phrase embeddings. Kalchbrenner and Blunsom (2013) utilize a simple convolution model to generate phrase embeddings from word embeddings. Mikolov et al. (2013) consider a phrase as an indivisible n-gram. Obviously, these methods of learning phrase embeddings either focus on some aspects of the phrase (e.g. reordering pattern), or impose strong assumptions (e.g. bagof-words or indivisible n-gram). Therefore, these phrase embeddings are not suitable to fully...

Citation Context

...yer predicts residuals, which the first layer fails to predict well. In our model, compression amounts to exploiting redundancies in the relations between successive sequence elements. In contrast to =-=[9]-=- who introduce a recursive bi-linear autoencoder for modeling language, our model is recurrent and trained to predict not reconstruct. The model by [10] is similar to our model in that it learns the d...

Citation Context

...target-dependent sentiment classification. RNN utilizes the recursive structure of text, and it has achieved state-of-the-art sentiment analysis results for movie review dataset (Socher et al., 2012; =-=Socher et al., 2013-=-). The recursive neural models employ the semantic composition functions, which enables them to handle the complex compositionalities in sentiment analysis. Specifically, we propose a framework which ...

Citation Context

...icting human judgements. 1 Introduction Word embeddings are popular representations for syntax (Turian et al., 2010; Collobert and Weston, 2008; Mnih and Hinton, 2007), semantics (Huang et al., 2012; =-=Socher et al., 2013-=-), morphology (Luong et al., 2013) and other areas. A long line of embeddings work, such as LSA and randomized embeddings (Ravichandran et al., 2005; Van Durme and Lall, 2010), has recently turned to ...

Citation Context

...-th character by the network with parameters θ. Like Collobert et al. (2011) and Zheng et al. (2013), our model is also trained at sentencelevel and carries out inference globally. 3.2 Tensor Neural Network A tensor is a geometric object that describes relations between vectors, scalars, and other tensors. It can be represented as a multi-dimensional array of numerical values. An advantage of the tensor is that it can explicitly model multiple interactions in data. As a result, tensor-based model have been widely used in a variety of tasks (Salakhutdinov et al., 2007; Krizhevsky et al., 2010; Socher et al., 2013b). In Chinese word segmentation, a proper modeling of the tag-tag interaction, tag-character interaction and character-character interaction is very important. In linear models, these kinds of interactions are usually modeled as features. In conventional neural network models, however, the input embeddings only implicitly interact through the non-linear function which can hardly model the complexity of the interactions. Given the advantage of tensors, we apply a tensor-based transformation to the input vector. Formally, we use a 3-way tensor V [1:H2] ∈ RH2×H1×H1 to directly model the interact...

Citation Context

...en as a generalization of the recurrent neural network [5], which has a specific type of skewed tree structure (see Figure 1). They have been applied to parsing [6], sentence-level sentiment analysis =-=[7, 8]-=-, and paraphrase detection [9]. Given the structural representation of a sentence, e.g. a parse tree, they recursively generate parent representations in a bottom-up fashion, by combining tokens to pr...