Efficient learning of sparse representations with an energy-based model

DMCA

Efficient learning of sparse representations with an energy-based model (2006)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Marc'Aurelio Ranzato , Christopher Poultney , Sumit Chopra , Yann Lecun

Venue:	ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS (NIPS 2006
Citations:	218 - 15 self

BibTeX

@INPROCEEDINGS{Ranzato06efficientlearning,
    author = {Marc'Aurelio Ranzato and Christopher Poultney and Sumit Chopra and Yann Lecun},
    title = {Efficient learning of sparse representations with an energy-based model},
    booktitle = {ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS (NIPS 2006},
    year = {2006},
    pages = {1137--1144},
    publisher = {MIT Press}
}

OpenURL

Abstract

We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces “stroke detectors ” when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps. 1

Keyphrases

energy-based model efficient learning sparse representation linear encoder encoder output two-phase em-like fashion expensive sampling stroke detector gabor-like filter topographical filter map code vector novel unsupervised method natural image patch optimal code convolutional network minimum-energy code vector overcomplete feature mnist dataset linear decoder input patch unsupervised method quasi-binary sparse code vector first layer handwritten numeral error rate

DMCA