Download:
by George D. Magoulas, Vassilis P. Plagianakos, George S. Androulakis, Michael N. Vrahatis, Department Of Informatics
http://www.math.upatras.gr/~vpp/pdf/a-2.pdf
Add To MetaCart
Abstract:
Abstract:- In this paper we propose a framework for developing globally convergent batch training algorithms with adaptive learning rate. The proposed framework provides conditions under which global convergence is guaranteed for adaptive learning rate training algorithms. To this end, the learning rate is appropriately tuned along the given descent direction. Providing conditions regarding the search direction and the corresponding stepsize length this framework can also guarantee global convergence for training algorithms that use a different learning rate for each weight. To illustrate the effectiveness of the proposed approach on various training algorithms simulation results are provided. Key-Words:- Global convergence, learning rate adaptation, batch training algorithms, steepest descent, feedforward neural networks. 1.
Citations
|
2140
|
Learning Internal Representations by Error Propagation
– Rumelhart, Hinton, et al.
- 1986
|
|
904
|
Practical optimization
– Gill, Murray, et al.
- 1981
|
|
599
|
Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall
– DENNIS, SCHNABEL
- 1983
|
|
371
|
A direct adaptive method for faster backpropagation learning: the RPROP algorithm
– Riedmiller, Braun
- 1993
|
|
352
|
Rheinboldt, Iterative Solution of Nonlinear Equations
– Ortega, W
- 2000
|
|
245
|
Increased Rates of Convergence Through Learning Rate Adaptation. Neural Networks
– Jacobs
- 1988
|
|
223
|
Faster-Learning Variations on Back-Propagation: An Empirical Study
– Fahlman
- 1988
|
|
159
|
A scaled conjugate gradient algorithm for fast supervised learning
– Møller
- 1993
|
|
94
|
Minimization of Functions Having Lipschitz Continuous First-Partial Derivatives
– Armijo
- 1966
|
|
91
|
Accelerating the convergence of the backpropagation method
– Vogl, Mangis, et al.
- 1988
|
|
73
|
Improving the convergence of back-propagation learning with second order methods
– Becker, LeCun
- 1989
|
|
66
|
Optimization: Algorithms and Consistent Approximations
– Polak
- 1997
|
|
58
|
Theory of algorithms for unconstrained optimization
– Nocedal
|
|
42
|
Global convergence properties of conjugate gradient methods for optimization
– GILBERT, NOCEDAL
- 1992
|
|
42
|
Convergence conditions for ascent methods
– Wolfe
- 1969
|
|
32
|
Acceleration Techniques for the Back propagation Algorithm
– Silva, Almeida
- 1990
|
|
30
|
Accelerated backpropagation learning: two optimization methods
– Battiti
- 1989
|
|
20
|
Convergence conditions for ascent methods. II: Some corrections
– Wolfe
- 1971
|
|
18
|
An adaptive training algorithm for back–propagation networks
– Chan, Fallside
- 1987
|
|
18
|
Automatic learning rate maximization by on-line estimation of the Hessian's eigenvectors
– Cun, Simard, et al.
- 1993
|
|
18
|
Improving the Convergence of the Backpropagation Algorithm Using Learning Rate Adaptation Methods
– Magoulas, Vrahatis, et al.
- 1999
|
|
13
|
A class of gradient unconstrained minimization algorithms with adaptive stepsize
– Vrahatis, Androulakis, et al.
- 2000
|
|
11
|
An analysis of premature saturation
– Lee, Oh, et al.
- 1993
|
|
10
|
An accelerated learning algorithm for multilayer perceptron networks
– Parlos, Fermandez, et al.
- 1994
|
|
10
|
Speeding–up backpropagation – A comparison of orthogonal techniques
– Pfister, Rojas
- 1993
|
|
9
|
Androulakis, “Effective back-propagation with variable stepsize
– Magoulas, Vrahatis, et al.
- 1997
|
|
7
|
Cauchy’s method of minimization
– Goldstein
- 1962
|
|
6
|
Rescaling of variables
– Rigler, Irvine, et al.
- 1991
|
