D.P. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Transaction on Automatic Control, 21(2):174--184, 1976.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....These results extend the early work of [25] 31] and [39, 40] where certain form of strong convexity is assumed. Our analysis is fairly general as it is applicable to a wide class of iterative descent algorithms, including a gradient projection algorithm of Goldstein [11] and Levitin and Polyak [2]; a certain matrix splitting algorithm ( 33, 36] coordinate descent methods (see [1, 5, 7, 22, 35, 42] the extragradient method of Korpelevich [16] the proximal minimization algorithm of Martinet [34] The rest of the paper is organized as follows. Section 2 presents the notions of ....

....and f(x k 1 ) Gamma f(x k ) Gamma 2 kx k Gamma x k 1 k 2 for some 2 0; 3.12) The condition (3.12) ensures sufficient descent at each iteration. The above scheme (3. 10) is a broad class that includes (a) a gradient projection algorithm of Goldstein [11] and Levitin and Polyak [2]; b) a certain matrix splitting algorithm ( 33, 36] coordinate descent methods (see [1, 5, 7, 22, 35, 42] c) the extragradient method of Korpelevich [16] d) the proximal minimization algorithm of Martinet [34] The reference [28] provides a detailed justification of why the above listed ....

D.P. Bertsekas, "On the Goldstein--Levitin--Polyak gradient projection method," IEEE Trans. Auto. Contr., 21 (1976), 174--184.

....on (1.16) are referred to as projected (semismooth) Newton methods. However, 1.16) may not hold when x # X intX, no matter how small # is. This is illustrated in Example 1.1. For any x # X, define the gradient direction dG by dG = ###(x) for some # 0. Also for any x # X and # # [0, 1] define the projected gradient direction dG (#) by dG (#) #X [x #dG ] x (1.17) and the projected Newton direction dN (#) by dN (#) #X [x #dN ] x. 1.18) Example 1.1 Consider a two dimensional NCP where X = x # R 2 x # 0 and F : R 2 # R 2 is a continuously ....

....Consider a two dimensional NCP where X = x # R 2 x # 0 and F : R 2 # R 2 is a continuously di#erentiable function with F (0, 0) # 1 2 0 # , F # (0, 0) # 0 1 1 1 # . Figure 1 displays the gradient direction ##, the projected gradient direction dG (#) # # [0, 1] and the Newton direction dN of # (# = 0) at x = 0, 0) respectively. It is clearly shown in Figure 1 that #X [x #dN ] x, # # # 0. This demonstrates that there is no theoretical guarantee of success for methods based on (1.16) despite the fact it has been widely used in various projected ....

[Article contains additional citation context not shown here]

D.P. Bertsekas, "On the Goldstein-Levitin-Polyak gradient projection method", IEEE Transactions on Automatic Control, 21 (1976), 174--184.

....contains the first p columns of A and where d = b Gamma Ax [0] 6) We also assume that the stopping rule parameters ATOL and BTOL are given, according to [23] Restricted LSQR Step 1 : Initialization. Given x [0] and J [0] compute y [0] and d from (4) and (6) Also define fi [1] u [1] d; ff [1] v [1] B t u [1] w [1] v [1] OE [1] fi [1] ae [1] ff [1] 7) and set i = 1. Step 2 : Minor iterations. Step 2.a : Bidiagonalization. Set fi [i 1] u [i 1] Bv [i] Gamma ff [i] u [i] ff [i 1] v [i 1] B t u [i 1] ....

....contains the first p columns of A and where d = b Gamma Ax [0] 6) We also assume that the stopping rule parameters ATOL and BTOL are given, according to [23] Restricted LSQR Step 1 : Initialization. Given x [0] and J [0] compute y [0] and d from (4) and (6) Also define fi [1] u [1] = d; ff [1] v [1] B t u [1] w [1] v [1] OE [1] fi [1] ae [1] ff [1] 7) and set i = 1. Step 2 : Minor iterations. Step 2.a : Bidiagonalization. Set fi [i 1] u [i 1] Bv [i] Gamma ff [i] u [i] ff [i 1] v [i 1] B t u [i 1] Gamma fi ....

[Article contains additional citation context not shown here]

D.P. Bertsekas, "On the Goldstein-Levitin-Polyak gradient projection method", IEEE Transactions on Automatic Control, vol. 21, pp. 174--184, 1976.

....non increasing. On the other hand, from (22) kx k ( x k k k rf(x k )k, so kx k ( x k k = O(1) 26) Thus, 25) implies lim 0 f(x k ) f(x k ( rf(x k ) T (x k ( x k ) 1: Hence the result (24) immediately follows. We prove (c) by giving an example from Bertsekas [1]: De ne k = m k 2 1 , where m k is the smallest integer such that (24) is satis ed. From (b) we know this k exists. Then by using this x k ( k ) as x C k , both (9) and (10) are satis ed. The following theorem from Calamai and Mor e [4] is very important to our convergence proof. ....

D. P. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Trans. Automat. Control, 21:174-184, 1976.

....to guess the set of bounds that are activeatthe solution. Such algorithms have been analyzed in the context of general nonlinear optimization by many authors: originally introduced by Goldstein #1964#, Levitin and Polyak #1966#, this class of algorithms has seen continued development #see Bertsekas, 1976 and Dunn, 1981, for example#. 28 linear least squares problems with bound constraints More recently, the concept of guessing the set of active bounds using a projected gradient step has been incorporated in e#cient trust region algorithms by Conn et al. #1988a#, Conn et al. #1988b#, and ....

Bertsekas, D. P. #1976#. On the Goldstein-Levitin-Polyak gradient projection method,IEEE Transactions on Automatic Control 21: 174#184.

....in [5] Therefore, it is natural and rather easy to transport the spectral gradient idea with a nonmonotone line search to the projected gradient case in order to speed up the convergence of the projected gradient method. In particular, in this work we extend the practical version of Bertsekas [2] that enforces an Armijo type of condition along the curvilinear projection path. This practical version is based on the original version proposed by Goldstein [16] and Levitin and Polyak [19] We also apply the new ingredients to the feasible continuous projected path that will be properly ....

....by Goldstein [16] and Levitin and Polyak [19] We also apply the new ingredients to the feasible continuous projected path that will be properly defined in Section 2. The convergence properties of the projected gradient method for different choices of stepsize have been extensively studied. See [2, 3, 7, 11, 16, 19, 22, 30], and other authors. For an interesting review of the different convergence results that have been obtained under different assumptions, see Calamai and Mor e [7] For a complete survey see Dunn [12] In Section 2 of this paper we define the spectral projected gradient algorithms and 2 we prove ....

D. P. Bertsekas [1976], On the Goldstein-Levitin-Polyak gradient projection method, IEEE Transactions on Automatic Control 21, pp. 174--184.

.... a group in the departments of Mathematics and Electrical and Computer Engineering at North Carolina State University to a variety of optimization problems that arise in computer aided design of microwave devices [17] 16] 19] 20] The algorithm is an extension of the projected gradient method [1] and as such is simple to implement and its performance can be improved by application of quasi Newton methods. The purpose of this theoretical paper is to analyze the convergence properties of the method. The algorithm discussed in this paper was designed for the specific applications described ....

....showed how the multidirectional search algorithm could be efficiently implemented on parallel processor computers. Rather than sample the variable space as a true filtering algorithm would, we propose use of a finite difference gradient based method, for example the projected gradient method [1], with the step size in the difference chosen as it would be if OE were floating point roundoff. For example, in the case of forward differences h p kOEk 1 . Since kOEk 1 is not known, we apply the finite difference gradient based method, decrease h after convergence, and apply the ....

[Article contains additional citation context not shown here]

D. B. Bertsekas, On the Goldstein-Levitin-Polyak gradient projection method, IEEE Trans. Autom. Control, (1976), pp. 174--184.

....hand, from (3.5) kx k (ff) Gamma x k k ffk rf(x k )k, so ff kx k (ff) Gamma x k k = O(1) 3.9) Thus, 3.8) implies lim ff 0 Gammaf (x k ) f(x k (ff) rf(x k ) T (x k (ff) Gamma x k ) 1: Hence the result (3.7) immediately follows. We prove (c) by giving an example from Bertsekas [ 1976 ] Define ff k = fl mk 2 fl 1 , where m k is the smallest integer such that (3.7) is satisfied. From (b) we know this ff k exists. Then by using this x k (ff k ) as x C k , both (2.5) and (2.6) are satisfied. 2 The following theorem from Calamai and Mor e [ 1987 ] is very important to our ....

D. P. Bertsekas. On the GoldsteinLevitin -Polyak gradient projection method. IEEE Trans. Automat. Control, 21:174--184, 1976.

....to avoid the local minima caused by oscillations or discontinuities in 6 . Implicit filtering is designed to solve problems in which the amplitude of the noise 6 decreases near the optimal point. Implicit filtering is a difference gradient implementation of the gradient projection algorithm [2] in which the difference increment is reduced in size as the iteration progresses. By using a sequence of difference increments, called scales, one hopes to jump over local minima and implicitly filter the noise. The algorithm was originally described and applied to problems in electrical ....

.... implicitly filter the noise. The algorithm was originally described and applied to problems in electrical engineering in [29] 28] 30] and [27] In order to fully describe the algorithm as used in this paper, we must set notation by first describing the gradient projection algorithm from [2]. Let : denote the projection onto , which is defined, for ; by : 2 # = A# if 2 #( B A# 2 # if #(CD 2 #1CB) # ) # if 2 #(E ) # (2.3) A gradient projection step transforms a current approximation GF to a local minimizer to a new ....

[Article contains additional citation context not shown here]

D. B. BERTSEKAS, On the Goldstein-Levitin-Polyak gradient projection method, IEEE Trans. Autom. Control, 21 (1976), pp. 174--184.

....and geosciences [1, 2, 11, 14] The algorithm used in IFFCO is analyzed in [10, 13] IFFCO and algorithms like IFFCO are applied to problems far more complex that those that satisfy the hypotheses of the theoretical results. IFFCO is a variation on the gradient projection method described in [3], that uses a sequence of finite difference steps (scales) to approximate the gradient. A brief outline of the algorithm used in IFFCO is given below. A more detailed description of the algorithm is given in Section 3.2. 2 Algorithm 1 BriefIFFCO Pick initial x and h; find f(x) and the Difference ....

....IFFCO is a projected quasi Newton algorithm that uses a decreasing sequence of finite difference steps (scales) to approximate the gradient. It uses an approximation to the Hessian and a line search algorithm that gives the code global capabilities. IFFCO is based on an algorithm presented in [3]. As mentioned in Chapter 1 the function to be minimized is of the form: f(x) f(x) OE(x) f(x) is a smooth function with only a few minima within the hyper box defined by the constraints. OE(x) contains the low amplitude high frequency terms of f(x) OE(x) creates local minima in f(x) ....

D. P. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Trans. Autom. Control, 21:174--184, 1976.

.... Gamma 1 fi fi fi fi L 2 kc Gamma c(ff)k 2 X hffiE (c) c Gamma (c(ff) i ff L 2 ; because of (A. 2) Next we show that the sequence of step sizes Gamma ff k Delta k2N generated by Algorithm 1 is bounded away from zero, which is an easy consequence of the following estimate [6] 8oe 2 (0; 1) 8ff 2 0; 2(1 Gamma oe) L : E(c) Gamma E(c(ff) oe hffiE (c) c Gamma c(ff)i ; 24 R. PINNAU AND A. UNTERREITER which is derived as follows E(c) Gamma E(c(ff) Z 1 0 hffiE (c t(c Gamma c(ff) c Gamma c(ff)i dt hffiE (c) c Gamma c(ff)i Gamma Z 1 0 jhffiE ....

D. P. Bertsekas, On the Goldstein--Levitin--Polyak gradient projection method, IEEE Trans. Autom. Contr., AC-21 (1976), pp. 174--184.

....discretization of elliptic partial differential operators and known properties of the spectrum of these operators leads to quantitative results on the effectiveness of the preconditioner. In x 4 we give numerical results based on problems from [8] and [2] We begin with the gradient projection [4] and projected Newton [5] methods for bound constrained optimization problems of the form min u2U f(u) 1.1) where f is twice Lipschitz continuously differentiable and U ae R N is given by U = fu j u low u u high g (1.2) Version of June 25, 1997. This research was supported by National ....

D. B. BERTSEKAS, On the Goldstein-Levitin-Polyak gradient projection method, IEEE Trans. Autom. Control, 21 (1976), pp. 174--184.

....must be used capable of avoiding entrapment in the local minimum. To this end, an implicit filtering algorithm, developed by Gilmore and Kelley[4] was employed. Figure 5 Cost Function Variation with Two System Parameters [3] Implicit filtering is an implementation of the gradient projection [5] algorithm which uses finite difference gradients with a sequence of difference steps to jump over local minima and avoid entrapment by high frequency low amplitude (and possible discontinuous) perturbations (which we will refer to as noise) of smooth objective functions. Such noise can arise, ....

D.B. Bertsekas, On the Goldstein-Levitin-Polyak Gradient Projection Method, IEEE Transactions on Automatic Control, 21 (1976), pp. 174-184.

.... Gamma 1 fi fi fi fi L 2 kc Gamma c(ff)k 2 X hffiE (c) c Gamma (c(ff) i ff L 2 ; because of (20) Next we show that the sequence of step sizes Gamma ff k Delta k2N generated by Algorithm 1 is bounded away from zero, which is an easy consequence of the following estimate [Ber76] 8oe 2 (0; 1) 8ff 2 0; 2(1 Gamma oe) L : E(c) Gamma E(c(ff) oe hffiE (c) c Gamma c(ff)i ; which is derived as follows E(c) Gamma E(c(ff) Z 1 0 hffiE (c t(c Gamma c(ff) c Gamma c(ff)i dt hffiE (c) c Gamma c(ff)i Gamma Z 1 0 jhffiE (c t(c Gamma c(ff) Gamma ....

D. P. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Trans. Autom. Contr., AC-21:174--184, 1976.

....gradient projection method generates sequences that satisfy (4.6) Hence, Theorem 4.5 implies that if the sequence fx k g generated by the gradient projection method converges to x , then (4.7) holds. Closely related convergence results for the gradient projection method are due to Bertsekas [1, 2], Gafni and Bertsekas [12, 13] and Dunn [9] However, in these papers it is not shown that (4.6) holds; instead, it is shown that any limit point of the sequence fx k g is stationary. Theorem 4.5 also has application to the class of trust region methods for bound constrained optimization problems ....

D. P. Bertsekas, On the Goldstein--Levitin--Polyak gradient projection method, IEEE Trans. Automat. Control, 21 (1976), pp. 174--184.

....lim k 1 (V k ) oe (S k ) kOEk S k oe (S k ) 0; 6) then any accumulation point of the simplices is a critical point of f s . 2. Convergence Results 2.1. Implicit Filtering Implicit filtering is a difference gradient implementation of the gradient projection algorithm [2] in which the difference increment is reduced in size as the iteration progresses. In this way the sim 6 D. M. Bortz, C. T. Kelley plex gradient is used directly. It was originally proposed in [23] 20] 24] for various problems in semiconductor modeling and analyzed in [12] In this paper ....

D. B. BERTSEKAS, On the Goldstein-Levitin-Polyak gradient projection method, IEEE Trans. Autom. Control, 21 (1976), pp. 174--184.

....value without becoming trapped in a local minimum before reaching an acceptable parameter set. THE OPTIMIZER: IMPLICIT FILTERING Implicit filtering, originally proposed in the context of computer aided design of semiconductors [8,9,10] is a generalization of the gradient projection algorithm of [11] in which derivatives are computed with difference quotients. The step sizes (called scales) in the difference quotients are changed as the iteration progresses with the goal of avoiding local minima that are caused by high frequency, low amplitude oscillations such as those seen in Figs. 4 and 5. ....

D.B. Bertsekas, On the Goldstein-Levitin-Polyak Gradient Projection Method, IEEE Transactions on Automatic Control, 21 (1976), pp. 174-184.

....algorithms based on this method are of great interest, for they allow to drop and add many constraints at each iteration. They are of course especially efficient as the set X is such that the projection can be easily computed. Among the existing step size rules for choosing a k in (1. 1) Bertsekas [1] proposed a generalization of the well known Armijo rule for an efficient and easily implementable variation of the gradient projection method. This type of rule has thereafter been studied by Gafni and Bertsekas [11] and by Dunn (see [10] for instance) Bertsekas [3] also analyzed projected ....

....rule has thereafter been studied by Gafni and Bertsekas [11] and by Dunn (see [10] for instance) Bertsekas [3] also analyzed projected Newton methods using Armijo type rules for the problem minff(x)jx 0g. More recently, Calamai and Mor e [4] generalized the Armijo type rule proposed by Bertsekas [1] and considered a gradient projection method that selects the step size a k using conditions on the model s decrease and the steplength that are in the spirit of Goldstein s rules (see [2] for instance) Trust region methods for nonlinear optimization problems, another important ingredient of the ....

[Article contains additional citation context not shown here]

D. P. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Transactions on Automatic Control, 21:174--184, 1976.

....this author was supported by the Volkswagen Stiftung and North Atlantic Treaty Organization grant #CRG 920067. POINTWISE PROJECTED NEWTON METHOD 2 This method utilizes again the simple projection but has the drawback that it does not always produce a descent in the objective function. Bertsekas [1] and [2] introduced for the finite dimensional case with simple constraints such as upper and lower bounds on the variables a projected Newton method which alleviated this problem. For H = R n let u = P(u c Gamma ff c D Gamma1 c rOE(u c ) 1.5) where D c is a properly chosen matrix such ....

....I is understood to be identically zero on A we may also regard F I as a map from X to Y . We will use F I as both a theoretical and computational tool. When used in computation we must have knowledge of u on A. This is analogous to identification of the active set in finite dimensional problems [1], 2] In the infinite dimensional setting considered in this paper, complete identification of A is not possible. However, as we will show in Section 4, a useful subset of A can be identified. The authors extended in [13] Bertsekas gradient projection method to constrained compact fixed ....

D. B. Bertsekas, On the Goldstein-Levitin-Polyak gradient projection method, IEEE Trans. Autom. Control, 21 (1976), pp. 174--184.

....that IFFCO can minimize. Applications in which functions of the form (1.2) arise are described in [4] 5] 6] and [7] The algorithm used in IFFCO is analyzed in [3] To obtain IFFCO, send e mail to paul latour.math.ncsu.edu. IFFCO is a variation on the gradient projection method described in [1], that uses a sequence of finite difference steps (scales) to approximate the gradient. A brief outline of the algorithm used in IFFCO is given below. A more detailed description of the algorithm is given in Section 3.2. Algorithm 1.1 Given an initial point x 2 Q, an initial scale h, and a lower ....

....IFFCO is a projected quasi Newton algorithm that uses a decreasing sequence of finite difference steps (scales) to approximate the gradient. It uses an approximation to the Hessian and a line search algorithm that gives the code global capabilities. IFFCO is based on an algorithm presented in [1]. As mentioned in Chapter 1 the function to be minimized is of the form: f (x) f(x) OE(x) f(x) is smooth function with only a few minima within the hyper box defined by the constraints. OE(x) contains the low amplitude high frequency terms of f(x) OE(x) creates local minima in f(x) These ....

D. B. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Trans. Autom. Control, (2):174--184, 1976.

No context found.

D.P. Bertsekas. On the Goldstein-Levitin-Polyak gradient projection method. IEEE Transaction on Automatic Control, 21(2):174--184, 1976.

No context found.

D. P. Bertsekas [1976], On the Goldstein-Levitin-Polyak gradient projection method, IEEE Transactions on Automatic Control 21, pp. 174-184.

No context found.

D. P. Bertsekas [1976], On the Goldstein-Levitin-Polyak gradient projection method, IEEE Transactions on Automatic Control 21, pp. 174--184.

No context found.

D. Bertsekas, On the Goldstein-Levitin-Polyak gradient projection method, IEEE Transactions on Automatic Control 21 (1976), pp. 174--184.

No context found.

Bertsekas, D.P. (1976), "On the Goldstein-Levitin-Polyak Gradient Projection Method", IEEE Transactions on Automatic Control, 21, 174-183.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC