Bourlard, H., & Kamp, Y. (1988). Auto-Association by Multilayer Perceptron and Singular Value Decomposition.. Biological Cybernetics, 58, 291--294.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... the gradient descent based learning rule as follows: 7) where is the time scale, is the diagonal matrix whose th element is the derivative of , i.e. The close relationship between autoencoders and principal component analysis has been discussed in many papers, for example, 16] 18] and [22]. For a linear autoencoder, i.e. it has been proven that the autoencoder acts like a linear PCA filter. In this case, the output reconstructs the input with a corresponding squared reconstruction error , which measures how well the model fits the data. Similarly, in a nonlinear autoencoder, the ....

H. Bourlard and Y. Kamp, "Autoassociation by multilayer perceptrons and singular value decomposition," Biol. Cybern., vol. 59, pp. 291--294, 1988.

....outperform the non linear one in terms of both training speed and compression performance. Most of the back propagation neural networks developed for image compression are, in fact, designed as linear [9,14,47] Theoretical discussion on the roles of the sigmoid transfer function can be found in [5,6]. With this basic back propagation neural network, compression is conducted in two phases: training and encoding. In the rst phase, a set of image samples is designed to train the network via the back propagation learning rule which uses each input vector as the desired output. This is ....

H. Bourlard, Y. Kamp, Autoassociation by multilayer perceptrons and singular values decomposition, Biol. Cybernet. 59 (1988) 291}294.

....been overcome. One of the main reasons why this worry about the opacity of learned NNs has been alleviated is that controlled experiments like auto encoders effectively convinced the research community that the hidden units of the NN were in fact finding useful, compressed versions of their inputs [1, 2, 6]. While we are confident that the PADO learning architecture, through evolutionary computation, does learn non trivial algorithms and can be applied to real world signals, it seems that a tangent is in order. This paper is that tangent. We will show, through a controlled experiment, using ....

H. Bourlard and Y. Kamp. Autoassociation by multilayer perceptrons and singular vlue decomposistion. In Biological Cybernetics., 1988.

....network with a bottleneck of size r has long been known to be the projection into the subspace spanned by the first r principal components of the training data. This is true if the training algorithm minimizes squared error at the output and achieves the global minimum of that error. Furthermore, Bourlard and Kamp (1988) have shown that if all layers of the network after the bottleneck are linear the optimal transformation remains unchanged even if nonlinear transfer functions are added to units before the bottleneck. In other words, the result still holds for networks that are merely output linear. These ....

Bourlard, H. and Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291--294.

....of a bottleneck, which forces the network to optimally encode the input vectors, thus performing information compression and dimensionality reduction. With a single hidden layer of linear units, this approach was shown by Bourlard and Kamp to be equivalent to principal component analysis (PCA) [9]. Consequently, more complex networks with non linearities can be seen as implementing some form of non linear PCA . Such an approach is proposed in Ref. 10] where a five layer perceptron feedforward network is used for data validation. This network can be viewed as two independent three layer ....

H. Bourlard and Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition, Biological Cybernetics 59 (1988), 291--294.

....can be well separated by the new transformed features. A common practice is linear transformation: y=A T X, where X is the original m vector, y is new l vector and A T is l x m transformation matrix. We can learn linear transformation matrix by neural network. The AutoAssociative Networks [2] [4] can be simply experessed as: input m nodes y y y y m output nodes hidden nodes w 12 is (i, j)th entry in matrix x x x x 1 2 3 m 1 2 3 m w Figure 1: The Auto Associative Networks ....

Bourland H., Kamp Y "Auto-association by multilayer perceptrons and singular value decomposition" Biological Cybernetics, Vol. 59, pp.291-294, 1988.

....dimension compressing hidden layer, r units Figure 1: General structure of AANN. An AANN with only linear units will perform linear autoassociation. A 3 layer linear AANN will perform PCA [3] Bourlard and Kamp analysed linear autoassociative networks based on singular value decomposition [8]. Baldi [9] gave a complete description of the error landscapes in terms of principal component analysis for linear AANNs. Bishop analysed AANNs in the context of dimensionality reduction. He stated that a 5layer network with nonlinear units in the hidden layers can perform better dimensionality ....

H. Bourlard and Y. Kamp, "Auto--association bymultilayer perceptrons and singular value decomposition," Biol. Cybernet.,vol. 59, pp. 291--294, 1988.

....Perceptron (MLP) can also perform dimensionality reduction if the number of units in the bottleneck layer is smaller than the dimensionality of the data. As for their capability of dimensionality reduction, it is shown that a three layer perceptron accomplishes equal reconstruction error as PCA[1, 2] and that nonlinear perceptrons with more (typically, five) layers attain smaller errors with the identical dimensionality[3, 4] In these cases, outputs of the bottleneck layer units are the parameters of the data representation. However, they do not compose an ordered representation. ....

Bourlard, H. and Kamp, Y., "Auto-association by multilayer perceptrons and singular value decomposition," Biol. Cybern., 1988.

....of a bottleneck, which forces the network to optimally encode the input vectors, thus performing information compression and dimensionality reduction. With a single hidden layer of linear units, this approach was shown by Bourlard and Kamp to be equivalent to principal component analysis (PCA) [9]. Consequently, more complex networks with non linearities can be seen as implementing some form of non linear PCA . Such an approach is proposed in Ref. 10] where a five layer perceptron feedforward network is used for data validation. This network can be viewed as two independent three layer ....

Bourlard, H. and Kamp, Y., Auto-association by multilayer perceptrons and singular value decomposition, Biological Cybernetics, 59, 291-294, 1988.

....network with a bottleneck of size r has long been known to be the projection into the subspace spanned by the first r principal components of the training data. This is true if the training algorithm minimizes squared error at the output and achieves the global minimum of that error. Furthermore, Bourlard and Kamp (1988) have shown that if all layers of the network after the bottleneck are linear the optimal transformation remains unchanged even if nonlinear transfer functions are added to units before the bottleneck. In other words, the result still holds for networks that are merely output linear. These ....

Bourlard, H. and Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291--294.

....Reduction Several researchers (e.g. Cottrell and Metcalfe 1991) have used layered feedforward auto associative networks with a bottle neck middle layer to perform dimension reduction. It is well known that auto associative nets with a single hidden layer cannot provide lower distortion than PCA (Bourlard and Kamp, 1988). Recent work (e.g. Oja 1991) shows that five layer auto associative networks can improve on PCA. These networks have three hidden layers (see Figure 1(a) The first and third hidden layers have non linear response, and are referred to as the mapping layers. The m n nodes of the middle or ....

H. Bourlard and Y. Kamp. (1988) Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291-294.

....by PCA is ill adapted. Further work must be done. Generalizations of this approach Many PCA generalizations have been proposed in order to take into account nonlinear phenomena. PCA like auto associative methods have been studied from the neural networks point of view with perceptron networks [2, 4, 24], but in fact these models remain linear. In this context, in [21] the model F ( x) d X k=1 h( k ; x ) k is used where h is a nonlinear function chosen arbitrarily. It can easily be shown that this model cannot represent more types of clouds X than PCA. A truly nonlinear ....

H. Boulard and Y. Kamp, " Autoassociation by multilayer perceptron and singular values decomposition", Biological Cybernetics, 59, 291-294, 1989.

....hidden units of an autoassociator, they project them on the Principal Components of the concept class data. While it has been sometimes claimed that even with nonlinearities in their hidden units, autoassociators do nothing other than to compute the principal components of the data in a domain ([Bourlard and Kamp1988], Cottrell and Munro1988] Chapter 5 will show that this claim is incorrect. 42 3.3 Threshold Determination In order to determine whether a novel test example is a positive or a negative instance of the concept, its reconstruction error must be compared against some threshold, dividing the ....

....autoassociators extract the first few principal components of a domain, or some close approximation to them and (2) that linear and nonlinear autoassociators are equivalently powerful on the task of image compression. These two results were theoretically confirmed by Baldi and Hornik (1989) and Bourlard and Kamp (1988). In regards to the first issue, Baldi and Hornik conducted a theoretical study of the L L autoassociator of figure 1(b) and showed that such networks actually project their input onto the subspace spanned by the first H principal components, where H corresponds to the number of hidden units. 4 ....

[Article contains additional citation context not shown here]

H. Bourlard and Y. Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291--294, 1988.

....input. Extending this idea, Oja s Subspace learning algorithm (Oja, 1989) finds a set of k weight vectors which span the same subspace as the first k principal components. The same is true of the weights learned by an auto encoder (Baldi and Hornik, chapter , this volume; Baldi and Hornik, 1989; Bourlard and Kamp, 1988). Sanger s Generalized Hebbian Algorithm (Sanger, 1989a; Sanger, 1989b) allows a group of linear units to learn a full principal components decomposition of the input distribution by combining Hebbian learning with Gram Schmidt orthogonalization. Finally, Foldiak (1990) has described a related, ....

Bourlard, H. and Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291--294.

.... claim, in their 1988 paper: for autoassociation with linear output units, the optimal weight values can be derived by standard linear algebra, consisting essentially in singular value decomposition (SVD) and making thus the nonlinear functions at the hidden layer completely unnecessary , [Bourlard and Kamp1988]. Bourlard and Kamp1988] backed their claim using theoretical considerations while [Cottrell and Munro1988] arrived at the same conclusion, independently, using an experimental methodology. Paradoxically, nonlinear autoassociators are also famous for their capacity to solve the encoder problem ....

.... ( for autoassociation with linear output units, the optimal weight values can be derived by standard linear algebra, consisting essentially in singular value decomposition (SVD) and making thus the nonlinear functions at the hidden layer completely unnecessary , Bourlard and Kamp1988] [Bourlard and Kamp1988] backed their claim using theoretical considerations while [Cottrell and Munro1988] arrived at the same conclusion, independently, using an experimental methodology. Paradoxically, nonlinear autoassociators are also famous for their capacity to solve the encoder problem ( Rumelhart et al..1986] a ....

[Article contains additional citation context not shown here]

H. Bourlard and Y. Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291--294, 1988.

....non linear autoassociators, deal with clustering very well [62] Concerning the learning process, however, the linear autoassociators behave significantly better. It has recently been shown [30] that linear autoassociators produce local minima free error surfaces, whereas, as pointed out in [63, 64], there is no such guarantee in the non linear case. Let us choose the quadratic cost function: E Q T : 1 2 T X t=1 E t = 1 2 T X t=1 n(L) X j=1 [x j(0) t) Gamma x j(L) t) 2 ; 25) and let N be an autoassociator with linear outputs. A general result can be derived by ....

H. Bourlard and Y. Kamp, "Auto-association by multilayer perceptrons and singular value decomposition, " Biological Cybernetics, vol. 59, pp. 291--294, 1988.

No context found.

Bourlard, H., & Kamp, Y. (1988). Auto-Association by Multilayer Perceptron and Singular Value Decomposition.. Biological Cybernetics, 58, 291--294.

No context found.

Bourlard, H., & Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59, 291-294.

No context found.

H. Bourlard and Y. Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291-294, 1988.

No context found.

Bourlard, H. & Kamp, Y. (1988), "Auto-association by multilayer Perceptrons and singular value decomposition ", Biol. Cyb. 59, 291-294.

No context found.

H. Bourlard and Y. Kamp, "Auto--association by multilayer perceptrons and singular value decomposition," Biol. Cybernet., vol. 59, pp. 291--294, 1988.

No context found.

H. Bourlard, Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition, Biological Cybernetics 59 (1988) 291-294.

No context found.

Bourlard, H & Kamp, Y (1988). Auto-association by Multilayer Perceptrons and Singular Value Decomposition. Biol. Cybernetics 59, 291-294.

No context found.

H. Bourlard and Y. Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59:291-294, 1988.

No context found.

H. Bourland and Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition, Biol. Cybern. 59, 291-294 (1988).

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC