P. Werbos, Backpropagation : Past and future, in: IEEE International Conferenmce of Neural Networks, Vol. 1, 1988, p. 343.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....Determining the appropriate size network remains an open problem [8] The problem of overfitting has received much attention in the literature. Methods of addressing this problem include using a holdout set to stop training early [19] crossvalidation [2] node pruning [7, 8] and weight decay [20], among others. These techniques approach optimal solutions given the inductive bias of the standard learning model, but do not consider possible enhancements to the inductive bias itself. Node pruning seeks to improve accuracy by simplifying network topology, rather than alleviating the problems ....

Werbos, P., "Backpropagation: Past and future", Proceedings of the IEEE International Conference on Neural Networks, IEEE Press, 1988, pp. 343-353.

....method for calculating the product of an n Thetan matrix with an arbitrary vector if the matrix happens to be the Hessian of a system whose gradient can be calculated in O(n) as is the case for most architectures encountered in practice. This fast Hessian vector product (Pearlmutter, 1994; Werbos, 1988; Mller, 1993) can be used in conjunction with (1) to create an efficient, iterative O(n) implementation of Newton s method. Unfortunately Newton s method has severe stability problems when used in nonlinear systems, stemming from the fact that the Hessian may be illconditioned and does not ....

P. J. Werbos. Backpropagation: past and future. In Proceedings of the IEEE International Conference on Neural Networks, San Diego, 1988, volume I, pages 343--353, Long Beach, CA, 1988. IEEE Press.

.... In output weight optimization backpropagation [18] OWO BP) linear equations are solved to find output weights and backpropagation is used to find hidden weights (those which feed into the hidden units) Unfortunately, backpropagation is not a very effective method for updating hidden weights [15,29]. Some researchers [11,16,17,20,31] have used the Levenberg Marquardt(LM) method to train the multilayer perceptron. While this method has better convergence properties [4] than the conventional backpropagation method, it requires O(N 2 ) storage and calculations of order O(N 2 ) where N is ....

P. Werbos, Backpropagation: Past and future, the Proceedings of the IEEE International Conference on Neural Networks, (1988) 343-353.

....out , can be given by = 2 ( 1 ) 2 ( 2 , 3 ( 3 ( u N p p i o i j w j j y (4) B. Output Weight Optimization Hidden Weight Optimization (OWO HWO) Training The backpropagation method traditionally used for training a feedforward neural network is not very efficient [8]. A more efficient updating algorithm is introduced in this section where the hidden weights are calculated by solving a set of linear equations [9] This would require that the desired hidden net functions be given, which is not normally the case. However, approximations of these desired net ....

P.Werbos, "Backpropagation: Past and Future," The Proceedings of the IEEE International Conference on Neural Networks, pp. 343-353 (1988).

.... applications are given in Biegler (1992) Chinchalkar and Coleman (1993) Coleman and Liao (1993) Coleman et al. 1992) Dunn (1993) Falk and McCormick (1986) Hager (1990) Jones (1967) Kunish and Sachs (1992) Liao (1993) McCormick (1972) McCormick and Sofer (1991) Schrady and Choe (1971) Werbos (1988) and Wu (1993) Finally, in a subject this complex, a single short article, necessarily, is only able to give an idea of the nature of the main issues in the current research. Moreover we have no doubt that our own particular biases show. Nevertheless we hope that the text and the references will ....

P. Werbos. Backpropagation: past and future. In Proceedings of the 2nd International Conference on Neural Networks. IEEE, New York, 1988.

....with many thousands connections, it is virtually impossible. Therefore, algorithms for automatic adjustment of weights have been developed. The process is called learning, since the network changes its behavior based on examples. Certainly the most famous learning algorithm is backpropagation [7] [8]. Although in the meantime more efficient procedures have been developed, the importance of backpropagation, conceptual as well as historical, remains unchanged. Our backpropagation applet has an appearance similar to the previous two. In addition, the user can select desired network response in ....

P.J. WERBOS, Backpropagation: Past and future, Proc. ICNN, I, (343-353) IEEE Press, New York, 1988

....The scaled conjugate gradient (SCG) algorithm circumvents the multiplication and computes this vector directly, without computing the Hessian. An elegant way to compute the exact product of a Hessian and an arbitrary vector has been independently rediscovered for neural networks in [6] 7] and [9]. This method is used for the numerical inversion proposed in this paper, with the exact values for and . We begin w ith the first order expansion of the error gradient, 10) Transposing Equation 10 and setting , w ith scalar and vector , gives . 11) The desired product is obtained by dividing ....

P.J. Werbos, "Backpropagation: Past and Future," Proc. IEEE Int. Conf. Neural Networks, Vol. 1, pp. 343-353, San Diego, CA, 1988.

....is one, well. Backpropagation can accommodate Page 11 both binary and continuous input or output. From a statistical viewpoint, backpropagation is the optimal supervised training method, as it converges to a nonlinear estimator with the maximum likelihood of being true (Movellan 1990, Werbos 1988). Backpropagation is readily available in software form, and increasingly, in hardware form. For these reasons backpropagation, or a variation of backpropagation, is usually the network of choice for pattern classification when teacher data is available. Figure 2. Typical Backpropagation Network ....

Werbos, Paul J., 1988, Backpropagation: Past and Future. Proceedings of the International Joint Conference on Neural Networks, I-343.

....baseline b ij . This could be different for each unit in a network and be used for individual tailoring of credit assignment. He suggests that informational connections could be added to a network to receive signals not affecting the output but calculating the baseline of a unit. Werbos [23] makes the noteworthy comment that these strategies leave out the crucial problem of maximizing some function f(r) of the accumulated reinforcement. The established theorems talk about the expected next reinforcement EfrjWg instead of the expected accumulated reinforcement Eff(r)jWg, which is ....

P. J. Werbos. Backpropagation: Past and future. In IEEE Int. Conf. on Neural Networks, pages 343--353, July 1988.

....4.1 Simple Backpropagation Networks Let us apply the above procedure to a simple backpropagation network, to derive the Rfbackpropg algorithm, a set of equations that can be used to efficiently calculate Hv for a backpropagation network. The Rfbackpropg algorithm was independently discovered by (Werbos, 1988, eq. 14) who derived it as a backpropagation process to calculate Hv = r w (v Delta r wE) where r wE is also calculated by backpropagation. Interestingly, that derivation is dual to the one given here, in that the direction of the equations is reversed, the backwards pass of the r wE algorithm ....

.... number, but if H is known to be positive definite, one can instead minimize x T Hx=2 x Delta b, which does not square the condition number (Press et al. 1988, page 78) This application of fast exact multiplication by the Hessian, in particular Rfbackpropg, was independently noted in (Werbos, 1988). 5.3 Step Size and Line Search Many optimization techniques repeatedly choose a direction v, and then proceed along that direction some distance , which takes the system to the constrained minimum of E(w v) Finding the value for which minimizes E is called a line search, because it ....

Werbos, P. J. (1988). Backpropagation: Past and future. In IEEE International Conferenceon Neural Networks, volume I, pages 343--353, San Diego, CA.

....exceed perceptrons in generalization to new examples. Index Terms Multi Layered Networks, learning environment, Backpropagation, pattern recognition, linearly separable classes. I. Introduction Supervised learning in Multi Layered Networks can be accomplished thanks to Backpropagation (BP ) [19, 25, 31]) Its application to several different subjects [25] and, particularly, to pattern recognition ( 3, 6, 8, 20, 27, 29] has been showing the power of this algorithm. A significant contribution to the understanding of these successful results has been given in [7, 15, 16, 22] In particular, in ....

....VI. II. Learning in Multi Layered Networks with Backpropagation. In this section, we define the formalism adopted throughout the paper and give a brief review of Backpropagation. Some details on the derivation of the algorithm and other general considerations can be found in several publications [15, 19, 25, 31]. Here, we propose a vectorial formulation of the basic equations which is particularly useful for our analysis of the convergence of BP . Basically, the problem of learning in MLNs is to find a set of weights which minimizes the mismatching between network outputs and target values. This ....

P. J. Werbos, "Back-Propagation: Past and Future," Proceedings of IEEE International Conference on Neural Networks, IEEE Press, Vol. I, pp. 343-353, New York, 1988.

....neural networks, which can be described as layered arrangements of linear adaptive units followed by nonlinear unimodal functions, typically contain a very large number of parameters that must be adapted. The most famous algorithm for adjusting these parameters is the backpropagation algorithm [62, 50, 63] which is nothing else than a generalization of LMS to this more complicated structure. Backpropagation suffers from the same convergence speed problems than LMS, but remedying to this problem turns out to be much more complicated in the case of neural networks, mostly because these circuits ....

P .J. Werbos. Backpropagation: Past and future. In Proc. of the IEEE Second Intl Conf. on Neural Networks, volume I, pages 333--353, San Diego, CA, July 1988.

.... Layer Perceptrons (MLP) The reference network in this report is the well known Multi Layer Perceptron (MLP) This network including the training algorithm called Backpropagation was first introduced by [23] Since then, a variety of different training algorithms has been developed (see, e.g. [26], 25] The technical report of L. Prechelt [18] uses the resilient propagation algorithm (RPROP) 21] This method performs a local adaptation of the weight updates according to the behaviour of the error function. This is achieved by an individual update value for each weight. This adaptive ....

P. J. Werbos. Backpropagation: Past and future. In Proc. of the ICNN-88, New York, pages 343--353, 1988.

No context found.

P. Werbos, Backpropagation : Past and future, in: IEEE International Conferenmce of Neural Networks, Vol. 1, 1988, p. 343.

No context found.

p. j. Werbos, "Backpropagation - Past and Future,"Proc. IEEE International Conference on Neural Networks,

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC