J. Nocedal, "Updating quasi-Newton matrices with limited storage," Math. Comp. 35 (1980) 773--782.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....a nonsmooth optimization algorithm described in Luksan and V1 cek [45] 46] Vlcek and Luksan [68] which is a hybrid algorithm that combines the characteristics of the variable metric method and the bundle method. We also apply a smooth optimization algorithm (L BFGS) described in Nocedal [54] and Liu and Nocedal [43] Both methods require the computation of a subgradient (respectively the gradient) of the cost functional. This subgradi ent (respectively gradient) is obtained from the adjoint model derived from the original numerical model. Accuracy tests for both the gradient and ....

....(to preserve the property of being bounded and other characteristics required for the global convergence) or the standard BFGS update after the descent steps (for both types of updates see Fletcher [16] 5. 2 L BFGS Unconstrained optimization algorithm We also tested the L BFGS method (Nocedal [54], Liu and Nocedal [43] Nocedal and Wright [55] which performs the unconstrained minimization of a smooth nonlinear function for which the gradient is available. L BFGS is a limited memory method based on the well known BFGS (Broyden Fletcher Goldfarb Shanno) algorithm. The main idea of this ....

J. Nocedal, Updating Quasi-Newton matrices with limited storage, Math. Cornput.151(35), 773(1980).

.... Object Identifier 10.1109 TNN.2003.809425 In order to reach this aim, several iterative schemes exploiting parts of the Hessian approximation were implemented (e.g. the diagonal or the block diagonal part [29] Among the latter algorithms, the BFGS (Limited memory BFGS) methods [1] 27] [30], 31] have been studied extensively. The BFGS algorithms update continuously a Hessian approximation by using the most recent second order information available in the form of the vectors , The rate of convergence of BFGS methods can be improved if more information (corresponding to a larger ....

....algebra . The main idea is to reduce the complexity per step to a small number of fast transforms diagonalizing the matrices of .In this way one obtains flops instead of the of BFGS. Also the space complexity is reduced, from to . Notice that the limited memory BFGS method BFGS) 1] 27] [30], 31] and the memory less OSS OSS methods [4] 24] 37] turn out to be particular cases of the algorithm (6) The main idea in BFGS method is to use second order BFGS information from the most recent iterations. In BFGS the matrix depends on a limited number of pairs , More precisely, we ....

J. Nocedal, "Updating quasi-Newton matrices with limited storage," Math. Comp., vol. 35, pp. 773--782, 1980.

....One drawback of the BFGS update of eqn. 27 is that it requires storage for a matrix of size N x N and a number of calculations of order O(N 2) 20. Although the available storage is less of a problem now than it was a decade ago (for a possible method to cope with limited storage, see for example [Nocedal 1980]) the computational problem still exists when N becomes of the order of one hundred or more. Fortunately, it is possible to kill two birds with one stone. In [Battiti 1989] it is shown that it is possible to use a secant approximation with O(N) computing and storage time that uses second order ....

Nocedal, J. Updating quasi-Newton matrices with limited storage. Mathematics of Computation 35, 773-782.

....it may be well suited for problems with many inequality constraints. It is also more e#cient when the number of variables remains small or medium, say, fewer than 500, because it updates n n matrices by a quasi Newton (qN) formula. For problems with more variables, limited memory BFGS updates [39] can be used, but we will not consider this issue in this paper. Our motivation is based on practical considerations. During the last 15 years much progress has been realized on IP methods for solving linear or convex minimization problems (see the monographs [29, 10, 38, 44, 23, 42, 47, 49] For ....

J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comput., 35 (1980), pp. 773--782.

....(4n locations) the latter being more usually preferred in practice. These can also be preconditioned in a manner similar to the quadratic case. Then there are also methods that use rather more storage, such as CONMIN (Shanno and Phua [20] 7n locations) the Limited Memory BFGS method (Nocedal [18]) 9n locations) the Truncated Newton method (Dembo, Eisenstat and Steihaug [7] and many others. Amongst all of these, steepest descent methods hardly rate a mention in a modern text book on optimization, even though the storage requirements are minimal (3n locations) Indeed, the ....

J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. of Comp., 35, (1980), pp. 773-782.

....the dimension m of the dual variable vector y is large, say, more than several thousand, it is usually impossible to store the entire m m BFGS matrix H in standard workstations. In such cases, we may replace the full BFGS quasi Newton matrix used in Step 1 BFGS by the limited memory BFGS update [24]. In their recent paper [20] Morales and Nocedal proposed to use the limited memory BFGS quasi Newton matrix for preconditioning the CG method. In our predictor procedure described in Section 3.3, we can employ the limited memory BFGS quasi Newton matrix for preconditioning the CG method (or the ....

....issues remain to be studied further towards more practically e#cient implementations for large scale problems. Among others, we need to explore the use of . sparse Cholesky factorization of the dual matrix variable S = m # p=1 A p y p Iw C, the limited memory quasi Newton BFGS method [20, 24]. Acknowledgment. The authors would like to thank Professor Hiroshi Yabe for valuable discussions and suggestions on the quasi Newton BFGS method. In particular, he brought the authors attention to the recent paper [20] on the limited memory quasi Newton BFGS method. Also the authors would like ....

J. Nocedal, "Updating quasi-Newton matrices with limited storage," Math. Comp. 35 (1980) 773--782.

....premium in computational effort. e e 1 1 2 2 l l l M(r) r l Figure 1: Error measure proposed by Huber (Huber , 1973) The upper part above # is the l 1 norm while the lower part is the l 2 norm. In the work reported here we have used a version of the Limited Memory BFGS algorithm (Nocedal , 1980) as implemented in the Hilbert Class Library (Gockenbach et al. 1999) Other nonlinear iterative optimizers could be used we have treated the same examples with nonlinear conjugate gradients (Fletcher , 1980) with somewhat reduced efficiency but comparable results. The second section of our ....

Nocedal, J., 1980, Updating quasi-Newton matrices with limited storage: Mathematics of Computation, 95 , 339--353.

....the choice of the starting or ending value of oe. The numerical results presented in the following section show, however, that even simple choices can lead to good results. The initial minimization (to find a point on the path) is accomplished with the limited memory BFGS algorithm of Nocedal ([17]) The predictor corrector algorithm is based on following the solution of rP ( X; oe) 0: Differentiating with respect to oe, we obtain the following ordinary differential equation: n r 2 E( X) oe 2 W T W o d X doe = Gamma2oeW T W X : 7 As discussed above, the ....

J. Nocedal, "Updating Quasi-Newton Matrices with Limited Storage," Mathematics of Computation, Vol. 35, pp.773--782, 1980.

....To demonstrate the feasibility of doing this, we have taken several popular algorithms for large scale problems and translated them into HCL. Here we discuss the general framework for algorithms, and two specific examples: Nocedal s limited memory BFGS algorithm for unconstrained minimization [19] and Sorensen s implicitly restarted Arnoldi method for largescale eigenvalue problems [23] We have defined HCL classes to represent the algorithms themselves. The reason for doing so is that complicated algorithms are often built up out of simpler algorithms; if these algorithms are objects, ....

....not by as much as one might expect) We give examples below. We also point out that the performance gap between C and Fortran, while it might never be eliminated, is decreasing (see [22] for example) 3. 1 The limited memory BFGS algorithm The limited memory BFGS algorithm, due to Nocedal [19], is a variant on the popular BFGS algorithm for unconstrained minimization. These are both quasi Newton methods that build increasingly good Hessian approximations as the iteration proceeds. Since the limited memory version defines the approximation to the inverse Hessian in terms of outer ....

J. Nocedal. Updating Quasi-Newton matrices with limited storage. Mathematics of Computation, 35:339--353, 1980.

....argue that they may have a significant impact on the performance of the resulting method. In this paper we address precisely this issue. In our study we compare two implementations of the limited memory method: a) SNOPT, developed by Gill, Murray and Saunders [4] b) L BFGS as described by Nocedal [5]. Even though SNOPT is mainly intended for solving constrained problems, it provides us with a state of the art environment suitable for our comparison. The organization of the paper is as follows. In section 2 we briefly review the BFGS method and set the notation. We also describe the main ....

....is discarded and its location in memory taken by the new pair. An initial matrix H that is updated using m pairs of the form fs i ; y i g m i=1 is denoted as H(m) This approximation allows the computation of the product H(m)rf(x k ) in 4mn O(m) floating point operations; see Nocedal [5]. The memory requirements are 2mn O(m) locations to hold the m pairs of vectors (s; y) In L BFGS the initial matrix H is chosen as the identity matrix scaled by the quantity fl k = s T k y k y T k y k : 9) 2.2 SNOPT The release 5.3 4 of SNOPT is an SQP code that makes use of a ....

J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comput., 35 773-- 782(1980).

....k of the inverse of the Hessian matrix r 2 g(y k ; or the Cholesky factorization B k = L k (L k ) T of an approximation B k of the Hessian matrix r 2 g(y k ; itself of the function g( Delta; at y = y k . We can also utilize the limited memory BFGS quasi Newton method [24] when the dimension m of the y space is large. In spite of the nice feature (a) it is not so easy to numerically approximate the minimizer y( for small values of the barrier parameter 2 R . This is because the condition number of the Hessian matrix r 2 g( Delta; gets worse rapidly as ....

....the inverse of the coefficient Hessian matrix r 2 g(y( with a reasonable accuracy. 5 This preconditioning technique is essential to make our predictor procedure more effective. The paper [21] proposes a preconditioning of the CG method with the use of the limited memory BFGS update [24]. In addition to the distinguished features (a) b) c) and (d) of our method, e) our method is a variation of primal dual interior point methods [1, 15, 18, 20, 27, etc. for SDPs. Among others, we mention that our method can be extended to more general class of linear optimization problems ....

[Article contains additional citation context not shown here]

J. Nocedal, "Updating quasi-Newton matrices with limited storage," Math. Comp. 35 (1980) 773--782.

....direction p k as Mg k , where k is the iteration step, g k is the gradient of the cost function at x k , and M is an approximation to the Hessian. The matrix M is a positive de nite matrix obtained by updating a prescribed diagonal matrix with a limited number (5 7) of quasi Newton corrections (Nocedal 1980; Liu and Nocedal 1988) Under the approximation of (12) the updating quasi Newton corrections gathered in both loops can be used as if they were all within the same loop. 34 6.3 4D Var experiments using the truncated Newton like incremental method To avoid the strong e ect of nonlinearities ....

Nocedal, J., 1980: Updating quasi-Newton matrices with limited storage. Math. of Computation, 35, 773-782.

No context found.

J. Nocedal, "Updating quasi-Newton matrices with limited storage," Mathematics of Computation 35 (1980), pp. 773-782.

....given the use of gradient projection in the step computation we believe analyses similar to those in [7] and [9] should be possible, and that zigzagging should only be a problem in the degenerate case. The Hessian approximations B k used in our algorithm are limited memory BFGS matrices (Nocedal [21] and Byrd, Nocedal and Schnabel [6] Even though these matrices do not take advantage of the structure of the problem, they require only a small amount of storage and, as we will show, allow the computation of the generalized Cauchy point and the subspace minimization to be performed in O(n) ....

J. Nocedal, "Updating quasi-Newton matrices with limited storage", Mathematics of Computation 35 (1980): 773-782.

No context found.

J. Nocedal, "Updating quasi-Newton matrices with limited storage", Mathematics of Computation 35 (1980), pp. 773-782. 17

....Limited Memory Methods Various limited memory methods have been proposed# some combine conjugate gradient and quasi Newton steps, and others are very closely related to quasi Newton methods. The simplest implementation, and perhaps the most efficient, is the limited memory BFGS method (L BFGS) [52]2 ]2 ] It is a line search method in which the search direction has the form d k = H k g k : 4.5) The inverse Hessian approximation H k ,which is not formed explicitly, is defined by a small number of BFGS updates. In the standard BFGS method, H k is updated at every iteration by means of the ....

J. Nocedal (1980). Updating quasi-Newton matrices with limited storage, Mathematics of Computation 35, pp. 773-782.

....that H k can be written as H k = k;m j H k (V k;m Delta Delta Delta V k;1 ) ae k;m i s k;m s k;m (V k;m 1 Delta Delta DeltaV k;1 ) ae k;m 1 s k;m 1 s k;m 1 (V k;m 2 Delta Delta DeltaV k;1 ) ae k;1 s k;1 s k;1 : 2. 5) A recursive formula described in [12] takes advantage of the symmetry of this expression to compute the product H k g(x k ) efficiently. The numerical performance of the limited memory method L BFGS is often very good in terms of total computing time. However the method suffers from two major drawbacks: it is not rapidly convergent, ....

J. Nocedal, "Updating quasi-Newton matrices with limited storage," Mathematics of Computation 35 (1980), pp. 773-782.

....(2. 10) ae 0 (V T j Gamma1 Delta Delta Delta V T 1 )s 0 s T 0 (V 1 Delta Delta Delta V j Gamma1 ) ae 1 (V T j Gamma1 Delta Delta Delta V T 2 )s 1 s T 1 (V 2 Delta Delta Delta V j Gamma1 ) ae j s j Gamma1 s T j Gamma1 : A recursive formula described in [Nocedal 1980] makes use of the structure of (2.10) to compute the product (2.4) in approximately 4jn floating point operations. The storage requirements for this computation are 2jn spaces to hold the set of correction pairs, and 2j spaces to hold the scalars ae j and intermediate results. It is now clear that ....

Nocedal, J. 1980. Updating quasi-Newton matrices with limited storage. Math. Comput 35, 773--782.

....We will assume that the gradient g of f is available but that computing the Hessian matrix is not possible. Two of the most effective algorithms for these types of problems are: i) Hessian free inexact Newton methods (HFN) 14, 10, 3] and (ii) limited memory quasiNewton methods, such as L BFGS [12, 5, 7]. By a Hessian free Newton method we mean a Newton method in which the Hessian matrix is not available, and where products of the Hessian times a vector are either be computed by automatic differentiation or approximated by finite differences. Inexact Newton methods that have access to the ....

J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comput., 35 (1980), pp. 773-782.

....# V T k 1 V T k m 2 # s k m 1 s T k m 1 (V k m 2 V k 1 ) # k 1 s k 1 s T k 1 . 2.3) Thus instead of forming H(m) we can store the scalars # i and the vectors s i , y i , i = k 1, k m, which determine the matrices V i . A recursive formula described in [13, 19] takes advantage of the symmetry in (2.3) to compute the product H(m)v for any vector v with only 4mn floating point operations. The so called L BFGS method described in [19, 12, 9] updates Hessian approximations as follows. We first choose a sparse (usually diagonal) initial Hessian ....

.... s i , y i , i = k 1, k m, which determine the matrices V i . A recursive formula described in [13, 19] takes advantage of the symmetry in (2.3) to compute the product H(m)v for any vector v with only 4mn floating point operations. The so called L BFGS method described in [19, 12, 9] updates Hessian approximations as follows. We first choose a sparse (usually diagonal) initial Hessian approximation H and define the first m approximations through (2.3) as H(1) H(m) At this stage the storage is full, and to construct the new Hessian approximation, we first delete the ....

J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comput., 35 (1980), pp. 773--782.

No context found.

J. Nocedal, "Updating quasi-Newton matrices with limited storage," Math. Comp. 35 (1980) 773--782.

No context found.

J. Nocedal, Updating quasi-Newton matrices with limited storage, J. Mathematical of Comp., V. 35, N. 151, pp.773-782.

No context found.

J. Nocedal. Updating quasi-Newton matrices with limited storage. Mathematical of Comp., V. 35,N. 151,pp.773-782.

No context found.

Nocedal, J., 1980: Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35, 773-782.

No context found.

Nocedal, J., 1980: Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35, 773-782.

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC