DMCA
Contracting auto-encoders: Explicit invariance during feature extraction (2011)
| Venue: | In Proceedings of the Twenty-eight International Conference on Machine Learning (ICML’11 |
| Citations: | 77 - 12 self |
BibTeX
@INPROCEEDINGS{Rifai11contractingauto-encoders:,
author = {Salah Rifai and Pascal Vincent and Xavier Muller and Xavier Glorot and Yoshua Bengio},
title = {Contracting auto-encoders: Explicit invariance during feature extraction},
booktitle = {In Proceedings of the Twenty-eight International Conference on Machine Learning (ICML’11},
year = {2011}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We present in this paper a novel approach for training deterministic auto-encoders. We show that by adding a well chosen penalty term to the classical reconstruction cost function, we can achieve results that equal or surpass those attained by other regularized autoencoders as well as denoising auto-encoders on a range of datasets. This penalty term corresponds to the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input. We show that this penalty term results in a localized space contraction which in turn yields robust features on the activation layer. Furthermore, we show how this penalty term is related to both regularized auto-encoders and denoising auto-encoders and how it can be seen as a link between deterministic and non-deterministic auto-encoders. We find empirically that this penalty helps to carve a representation that better captures the local directions of variation dictated by the data, corresponding to a lower-dimensional non-linear manifold, while being more invariant to the vast majority of directions orthogonal to the manifold. Finally, we show that by using the learned features to initialize a MLP, we achieve state of the art classification error on a range of datasets, surpassing other methods of pretraining.
Keyphrases
feature extraction penalty term explicit invariance localized space contraction non-deterministic auto-encoders jacobian matrix frobenius norm encoder activation local direction turn yield robust feature learned feature penalty term result classical reconstruction cost function novel approach vast majority activation layer lower-dimensional non-linear manifold art classification error regularized autoencoders deterministic auto-encoders
