DMCA
A scaled conjugate gradient algorithm for fast supervised learning (1993)
| Venue: | NEURAL NETWORKS |
| Citations: | 451 - 0 self |
BibTeX
@ARTICLE{Møller93ascaled,
author = {Martin F. Møller},
title = {A scaled conjugate gradient algorithm for fast supervised learning},
journal = {NEURAL NETWORKS},
year = {1993},
volume = {6},
number = {4},
pages = {525--533}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
A supervised learning algorithm (Scaled Conjugate Gradient, SCG) with superlinear convergence rate is introduced. The algorithm is based upon a class of optimization techniques well known in numerical analysis as the Conjugate Gradient Methods. SCG uses second order information from the neural network but requires only O(N) memory usage, where N is the number of weights in the network. The performance of SCG is benchmarked against the performance of the standard backpropagation algorithm (BP) [13], the conjugate gradient backpropagation (CGB) [6] and the one-step Broyden-Fletcher-Goldfarb-Shanno memoryless quasi-Newton algorithm (BFGS) [1]. SCG yields a speed-up of at least an order of magnitude relative to BP. The speed-up depends on the convergence criterion, i.e., the bigger demand for reduction in error the bigger the speed-up. SCG is fully automated including no user dependent parameters and avoids a time consuming line-search, which CGB and BFGS uses in each iteration in order to determine an appropriate step size. Incorporating problem dependent structural information in the architecture of a neural network often lowers the overall complexity. The smaller the complexity of the neural network relative to the problem domain, the bigger the possibility that the weight space contains long ravines characterized by sharp curvature. While BP is inefficient on these ravine phenomena, it is shown that SCG handles them effectively.
Keyphrases
scaled conjugate gradient algorithm neural network problem dependent structural information scaled conjugate gradient optimization technique conjugate gradient method sharp curvature problem domain convergence criterion supervised learning algorithm appropriate step size speed-up depends weight space ravine phenomenon standard backpropagation algorithm memory usage one-step broyden-fletcher-goldfarb-shanno memoryless quasi-newton algorithm overall complexity user dependent parameter long ravine second order information conjugate gradient backpropagation numerical analysis neural network relative superlinear convergence rate
