B.T. Poljak. Introduction to Optimization. Optimization Software Inc., New York, 1987.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... same vector x , that is, i=1, m, 4) so that the iteration consisting of a cycle over the entire data set starting from x has the form ) x #f(x ) 5) Incremental methods are supported by stochastic convergence analyses [PoT73] Lju77] KuC78] TBA86] [Pol87], BeT89] Whi89] Gai94] BeT96] as well as deterministic convergence analyses [Luo91] Gri94] LuT94] MaS94] Man93] Ber95a] BeT96] It has been experimentally observed that the incremental gradient method (2) 3) often converges much faster than the steepest descent method (5) when ....

.... If # is instead taken to be a small constant, an oscillation within each data cycle arises, as shown by [Luo91] By contrast, for convergence of the steepest descent method, it is su#cient that the stepsize # is a small constant (this requires that be Lipschitz continuous, see e.g. [Pol87]) The asymptotic convergence rate of steepest descent with a constant stepsize is typically linear and much faster than that of the incremental gradient method. The behavior described above is most vividly illustrated in the case of a linear least squares problem where the vector x is ....

[Article contains additional citation context not shown here]

Poljak, B. T, "Introduction to Optimization," Optimization Software Inc., N.Y., 1987.

....2 1. Introduction where #m is obtained at the last step of the recursion # i = # i 1 # k #g i (# i 1 )g i (# i 1 ) i = 1, m, # k is a positive stepsize, and #0 = x k . Backpropagation methods are often e#ective, and they are supported by stochastic [PoT73] Lju77] KuC78] [Pol87], BeT89] Whi89a] Gai93] as well as deterministic convergence analyses [Luo91] Gri93] LuT93] MaS94] Man93] The main di#erence between stochastic and deterministic methods of analysis is that the former apply to an infinite data set (one with an infinite number of data blocks) ....

....(29) can be written approximately as x k 1 # x k 1 k 1 #g(x k )#g(x k ) # 1 #g(x k )g(x k ) 31) that is, as an approximate Gauss Newton iteration with diminishing stepsize. Thus, based on generic properties of gradient methods with diminishing stepsize (see e.g. [Pol87]) we can expect convergence to stationary points of the least squares problem, and a sublinear convergence rate. When # 1, the matrix H 1 i generated by the EKF recursion (22) will typically not diminish to zero, and x k may not converge to a stationary point of P m i=1 # m i #g i ....

[Article contains additional citation context not shown here]

Poljak, B. T, "Introduction to Optimization," Optimization Software Inc., N.Y., 1987.

.... ; A3) rf(x) x x ) jjrf(x)jj 2 ; A4) f(x) f jjx x jj 1 for all x such that f(x) f(x 0 ) where 1; and M; m; are positive constants. If f is a convex function satisfying condition (2) then assumption (A3) is ful lled for any L 1 and x 2 R n ; see, e.g. [16,18]. As the stopping criterion for the choice of nite T , we take the condition (5) jjrf l (ex(T ) jj ; where e x(t) t 0 is the numerical solution of system (3) obtained on a computer, rf l denotes a computer representation of the vector function rf with l binary digits for mantissa of ....

....kx(t) x k 1 1 (t) 1 1 ; and (15) krf(x(t) k M 1 (t) 1 ; t 0 hold. The proof readily follows from (12) Remark 2. Estimates for kx(t) x k and krf(x(t) k can easily be obtained without assumption (A4) In particular, the following inequality, see [16], can be established: krf(x)k 2 2 L(f(x) f ) i.e. krf(x(t) k p 2L (x(t) Using (12) in the last inequality, we obtain krf(x(t) k p 2L (t) 1 2(1 ) and kx(t) x k (m 1 p 2L) 1 (t) 1 2 (1 ) t 0: However, these estimates, as is easily ....

[Article contains additional citation context not shown here]

B. T. Poljak. Introduction to optimization, NAUKA Publishers, Moscow, 1983.

.... , m, 4) so that the iteration consisting of a cycle over the entire data set starting from x k has the form x k 1 = x k # k m X i=1 #f i (x k ) x k # k #f(x k ) 5) Incremental methods are supported by stochastic convergence analyses [PoT73] Lju77] KuC78] TBA86] [Pol87], BeT89] Whi89] Gai94] BeT96] as well as deterministic convergence analyses [Luo91] Gri94] LuT94] MaS94] Man93] Ber95a] BeT96] It has been experimentally observed that the incremental gradient method (2) 3) often converges much faster than the steepest descent method (5) when ....

.... If # k is instead taken to be a small constant, an oscillation within each data cycle arises, as shown by [Luo91] By contrast, for convergence of the steepest descent method, it is su#cient that the stepsize # k is a small constant (this requires that #f be Lipschitz continuous, see e.g. [Pol87]) The asymptotic convergence rate of steepest descent with a constant stepsize is typically linear and much faster than that of the incremental gradient method. The behavior described above is most vividly illustrated in the case of a linear least squares problem where the vector x is ....

[Article contains additional citation context not shown here]

Poljak, B. T, "Introduction to Optimization," Optimization Software Inc., N.Y., 1987.

....a principal aim of our work has been to avoid any type of boundedness assumption. For example, the classical analysis of Poljak and Tsypkin [PoT73] under essentially the same conditions as ours, shows that if f is bounded below, then f(x t ) converges and lim inf t## ##f(x t )# = 0 (see Poljak [Pol87], p. 51) The analysis of Gaivoronski [Gai94] for stochastic gradient and incremental gradient methods, under similar conditions to ours shows that lim t## ##f(x t )# = 0, but also assumes that f(x) is bounded below and that ##f(x)# is bounded over # n . The analysis of Luo and Tseng ....

Poljak, B. T., 1987. "Introduction to Optimization," Optimization Software Inc., N.Y.

No context found.

B.T. Poljak. Introduction to Optimization. Optimization Software Inc., New York, 1987.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC