Yann LeCun, Patrice Y. Simard, Barak Pearlmutter  Home/Search
Context Related
| unm.edu/~bap/paper...lecunnofigs.ps.gz microsoft.com/~patrice/PS/eigen.ps Cached: PS.gz PS PDF This document uses CoBlitz to cache paper downloads. If your firewall is blocking outgoing connections to port 3125, you can use these links to download local copies.
Image Update HelpPS.gz PS PDF From: unm.edu/~bap/publications (more) (Enter author homepages) |
Rate this article:      (best)Comment on this article |
Abstract: We propose a very simple, and well principled way of computing the optimal step size in gradient descent algorithms. The on-line version is very efficient computationally, and is applicable to large backpropagation networks trained on large data sets. The main ingredient is a technique for estimating the principal eigenvalue(s) and eigenvector(s) of the objective function's second derivative matrix (Hessian), which does not require to even calculate the Hessian. Several other applications of... (Update)
Context of citations to this paper: More
...H t Delta t . The only constraint on 0 is that 0 1=max . We use the on line algorithm developed by LeCun, Simard, and Pearlmutter [13] to find the largest eigenvalue prior to the start of training. 3 Examples In the following two subsections we examine the behavior of...
.... manipulation of the full Hessian is too expensive in computation and storage for FNNs with several hundred weights [3] Le Cun [11] proposed a technique, based on appropriate perturbations of the weights, for estimating on line the principle eigenvalues and eigenvectors of...
Cited by: More
Deterministic Nonmonotone Strategies for Effective.. - Plagianakos.. (Correct)
Speaker Normalization Improvement by Neural.. - Autiero.. (1999) (Correct)
Nonmonotone Methods for Backpropagation Training with.. - Plagianako, Vrahatis (Correct)
Active bibliography (related documents): More All
0.8: Automatic Learning Rate Maximization by On-Line.. - LeCun, Simard.. (1993) (Correct)
0.2: Static Versus Dynamic Sampling for Data Mining - John (1996) (Correct)
0.2: Object Oriented Design of a BP Neural Network Simulator and.. - Adamo, Anguita (1994) (Correct)
Similar documents based on text: More All
0.3: Reverse TDNN: An Architecture for Trajectory Generation - Simard, Le Cun (Correct)
0.3: Convolutional Networks for Images, Speech, and Time-Series - LeCun, Bengio (1995) (Correct)
0.3: Learning Prototype Models for Tangent Distance - Hastie, Simard, Säckinger (1995) (Correct)
Related documents from co-citation: More All
4: Increased Rates of Convergence Through Learning Rate Adaption (context) - Jacobs - 1988
4: Learning Internal Representations by Error Propagation (context) - Rumelhart, Hinton et al. - 1986
3: Numerical Methods for Unconstrained Optimization and Nonlinear Equations (context) - Dennis, Schnabel - 1983
BibTeX entry: (Update)
Y. LeCun, P. Y. Simard, and B. Pearlmutter. Automatic learning rate maximization by on-line estimation of the hessian's eigenvectors. In Giles, Hanson, and Cowan, editors, Advances in Neural Information Processing Systems, vol. 5, San Mateo, CA, 1993. Morgan Kaufmann. http://citeseer.ist.psu.edu/article/lecun92automatic.html More
@inproceedings{ lecunautomatic,
author = "Yann le Cun and Patrice Y. Simard and Barak A. Pearlmutter",
title = "Automatic Learning Rate Maximization by On-Line Estimation of the {H}essian's Eigenvectors",
pages = "156--163",
url = "citeseer.ist.psu.edu/article/lecun92automatic.html" }
Citations (may not include all citations):373 Adaptive Signal Processing (context) - Widrow, Stearns - 1985
162 Increased Rates of Convergence Through Learning Rate Adaptat.. (context) - Jacobs - 1987
144 Optimal Brain Damage - Le Cun, Denker et al.
45 Handwritten digit recognition with a back-propagation networ.. - Le Cun, Boser et al.
37 Improving the Convergence of Back-Propagation Learning with .. (context) - Becker, Le Cun - 1988
10 Eigenvalues of covariance matrices: application to neural-ne.. (context) - Le Cun, Kanter et al. - 1991
5 supervised learning on large redundant training sets (context) - Moller - 1992
2 Phd thesis (context) - Pearlmutter - 1993
1 Modeles connexionnistes de l'apprentissage (context) - Computer, COINS-TR- et al. - 1987
Documents on the same site (http://www.cs.unm.edu/~bap/publications.html): More
Relating Egomotion and Image Evolution - Pearlmutter, Gurvits (1995) (Correct)
VC Dimension of an Integrate-and-Fire Neuron Model - Zador (1996) (Correct)
Gradient Calculations for Dynamic Recurrent Neural Networks: A.. - Pearlmutter (1995) (Correct)
Online articles have much greater impact More about CiteSeer.IST Add search form to your site Submit documents Feedback
CiteSeer.IST - Copyright Penn State and NEC