D.B. Parker. Learning logic. Technical Report 47, Center for Computational Research in Economics and Management Science, MIT, April 1985.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....slogan: Patter sets do ot form vector spaces . Figure 2: A Feedforward Network layers and an output layer; each layer contains one or more Perceptton like processing units, for example Figure 2. The outputs of the previous layer are fed forward to the inputs of the next layer. In [Wrbos, 1974; Parker, 1985; Rumelhart and McClelland, 1986] an algorithm often called Back Error Propagation (BEP) is derived which trains feed forward net works to classify certain pattern classes, but this algorithm only works with activation functions which are continuous in the first derivative. One of the most ....

....investigate the relevant extensions of neural networks. The Back Propagation algorithm. This chapter deals with back error propagation (BEP) for Multi Layer Feed Forward networks defined over Clifford algebras. The case for real networks has been presented in [Bryson and Ho, 1969; Wrbos, 1974; Parker, 1985; Rumelhart and McClelland, 1986] The complex case has been treated by many people, see for instance [Little ct al. 1990; Henseler and Braspenning, 1990; Leung and Haykin, 1991; Benvenuto and Piazza, 1992] The account here of Clifford back propagation is a generalization of Georgiou and ....

D.B. Parker. Learning logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, Cambridge, M.A., 1985.

....of decision rules was again associated with the construction of linear hyperplanes in some space. An algorithm that allows for all weights of the neural network to adapt in order locally to minimize the error on a set of vectors belonging to a pattern recognition problem was found in 1986 [12, 13, 10, 8] when the back propagation algorithm was discovered. The solution involves a slight modification of the mathematical model of neurons. Therefore, neural networks implement piece wise linear type decision functions. In this article we construct a new type of learning machines, the so called ....

D. B. Parker. Learning logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, Cambridge, MA, 1985.

....approach to substitute it completely by a genetic algorithm. By means of some benchmark applications characteristic properties of both the genetic algorithm and the neural network are explained. 1 Introduction Multiple layer perceptrons (MLP) 1] commonly trained with backpropagation (BP) 2] [3] are very frequently used to solve a great variety of real world problems. Among the group of supervised trained networks this paradigm can be considered meanwhile as the standard architecture. That s why numerous extensions and modifications have been suggested to improve the results or to ....

D.B. Parker, Learning-logic, Report No. TR47, Massachusetts Inst. of Technology, Center for Computational Research in Economics and Management Science, 1985.

.... have been proposed, the most popular of which is error back propagation, discovered in the late sixties by Bryson and Ho [16] and rediscovered by Werbos in 1974 [83] It was, once again, independently rediscovered and popularized in the early eighties by several authors (Le Cun, 45] Parker, [60]; Rumelhart, Hinton and Williams [72] with these latter authors being its most ardent proselytes) An error back propagation network is composed of at least three layers of cells: an input layer, one or more hidden layers, and an output layer. In this introduction, I assume for A neural network ....

Parker D.B., Learning logic. Technical report TR-47, Center for Computational Research in Economics and Management Science, (Massachusetts Institute of Technology, Cambridge MA, 1985).

....genetic algorithm and the modified neural network are explained. KEY WORDS Artificial Neural Networks, Genetic Algorithms, Multiple Layer Perceptron, Backpropagation, Gradient Descent. 1 INTRODUCTION Multiple layer perceptrons (MLP) 1] commonly trained with backpropagation (BP) algorithm [2] [3] are very frequently used to solve a lot of real world problems. Among the supervised trained networks this paradigm can be considered meanwhile as the standard architecture [4] That s why numerous extensions and modifications have been suggested to improve the results or to achieve some required ....

D.B. Parker, Learning-logic, Report No. TR47, Massachusetts Institute of Technology, Center for Computational Research in Economics and Management Science, 1985. 1. Trained in standard configuration with momentum term. 825

....disciplines and has been developed by several di erent research groups. As pointed out by le Cun [38] to some extent, the basic elements of the theory can be traced back to the famous book of Bryson and Ho[9] A more explicit statement of the algorithm has been proposed by Werbos [56] Parker [43], le Cun [36] and members of the PDP group [44] Although many researchers have contributed in di erent ways in the development and proposition of di erent aspects of Backprop, there is no question that Rumelhart and the PDP group have the credit for the current high di usion of the algorithm. As ....

....parameters like learning rate and momentum in Backprop may a ect signi cantly the convergence, but may also hidden the essence of the problem. 3 Backprop: a brief review and notation An introduction on Backprop s theory can be found in numerous books and journals, under di erent points of view [26, 36, 43, 44, 56, 57]. In this section we review Backprop s basic principles and introduce a vectorial formulation of the equations which will turns out to be very useful for investigations on the local minima. Basically, the problem of learning in MLNs is to nd a set of weights which minimizes the mismatching ....

D.B. Parker, \Learning Logic", Technical Report TR-47, Center for Computational Research in Economics and Management Science, MIT, April 1985.

....processors (i.e. data parallelism) and with perfect load balancing. See Section 3 for discussions. When applied to the training of a nonlinear feedforward neural network, the above algorithm reduces to the popular backpropagation algorithm, as conceived by Werbos [Wer74] Le Cun [LeC85] Parker [Par85], and Rumelhart et al. RHW86] with training learning rates identified with stepsizes) In particular, the above algorithm with m = 1 (respectively, m = p) gives rise to the batch (respectively, pattern) backpropagation algorithm (see the discussions in [Wer90] and, with X a box, gives rise ....

D. B. Parker, Learning-logic, Center for Computational Research in Economics and Management Science Report No. TR-47, Massachusetts Institute of Technology, Cambridge, (1985).

....gradient) information about the training error is utilized. To speed up the training process, several second order optimization algorithms, which take advantage of second derivative (or Hessian matrix) information, have been proposed for training perceptrons [14] For example, the Newton method [26], the Quasi Newton method [32] the conjugate gradient method [30] as well as the Gauss Newton method [21] which is also used in the PPL [8] The fixed nonlinear activation (sigmoid) is a monotone nondecreasing differentiable function with very simple first derivative form, and possesses nice ....

D. B. Parker. Learning logic. Technical Report No. 47, Center for Computational Research in Economics and Management Science, MIT, April 1985.

....Training of a neural network is basically the search of an optimal weight set that minimizes a defined error (i.e. mean square error) between the network s results and the desired results. The most popular used training methods for feed forward networks are the back propagation algorithm [11][20] 31] and its variants [1] These algorithms give a prescription for changing the weights in any feed forward network to learn a training set of input output pairs. Mathematically, the basis is simply gradient descent. These learning algorithms perform a forward and a backward calculation ....

B. Parker. Learning Logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, Cambridge, MA, 1985.

....single obstacle to the widespread use of connectionist learning networks in real world applications is the slow speed at which the current algorithms learn. At present, the fastest learning algorithm for most purposes is the algorithm that is generally known as back propagation or backprop [6, 7, 9, 18]. The back propagation learning algorithm runs faster than earlier learning methods, but it is still much slower than we would like. Even on relatively simple problems, standard back propagation often requires the complete set of training examples to be presented hundreds or thousands of times. ....

Parker, D. B. Learning-Logic. Technical Report TR-47, Massachusetts Institute of Technology, Center for Computational Research in Economics and Management Science, Cambridge, MA, 1985.

....gradient) information about the training error is utilized. To speed up the training process, several secondorder optimization algorithms, which take advantage of second derivative (or Hessian matrix) information, have been proposed for training perceptrons [11] For example, the Newton method [19], the Quasi Newton method [25] the conjugate gradient method [23] as well as the GaussNewton method [15] Instead of training all the weights associated with all hidden units simultaneously as does in most BPLNs, a CCLN [5] incrementally train the weights associated each new candidate unit by ....

D. B. Parker. Learning logic. Technical Report No. 47, Center for Computational Research in Economics and Management Science, MIT, April 1985.

.... have been proposed, the most popular of which is error back propagation, discovered in the late sixties by Bryson and Ho [16] and rediscovered by Werbos in 1974 [83] It was, once again, independently rediscovered and popularized in the early eighties by several authors (Le Cun, 45] Parker, [60]; Rumelhart, Hinton and Williams [72] with these latter authors being its most ardent proselytes) An error back propagation network is composed of at least three layers of cells: an input layer, one or more hidden layers, and an output layer. In this introduction, I assume for simplicity, but ....

Parker D.B., Learning logic. Technical report TR-47, Center for Computational Research in Economics and Management Science, (Massachusetts Institute of Technology, Cambridge MA, 1985).

.... W 1 2 p X k=1 [Y (W ; R k ) Gamma S k ] T [Y (W ; R k ) Gamma S k ] 29) The Back Propagation algorithm, the most popular procedure to compute the the non zero elements of W which minimize the above error, was invented by Paul Werbos [79] in 1974, and re invented by David Parker [68] in 1982, being popularized by Hinton and Rumelhart [74] It is a gradient technique that slowly moves the weights towards their correct value. 4.3.1 Back Propagation Learning Algorithm Assume that the multi layer network is composed of M layers, such that the outputs of one layer are connected ....

....the changes in the weights becoming smaller, and smaller as the solution is approximated. The price to pay for using it, is the increased time and storage requirements. As it should be clear by now, every weight will have its own learning rate, thus twice as much memory will be required. Refer to [41, 68, 69] for a more detailed discussion on the subject. Yet another modification finds its motivation in the Widrow Hoff LMS algorithm presented earlier. Known as the on line version of backpropagation, it consists of dropping the summation over all patterns in the error derivative of equation (38) and ....

David B. Parker. Learning logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science - M.I.T., April 1985.

.... further studied by Pappas [15] and Bertsekas [2] When applied to neural network training, this method reduces to the very popular on line backpropagation algorithm with a momentum term (with training learning rates identified with stepsizes) as conceived by Werbos [20] Le Cun [11] Parker [16], and Rumelhart, Hinton, and Williams [18] see the discussions in [21] Numerical experience suggests that it is typically beneficial to choose # 0. In [18, p. 330] a value of # # .9 is recommended. An interesting one parameter generalization of this method for # = 0 and of the steepest ....

D. B. PARKER, Learning-Logic, Center for Computational Research in Economics and Management Science Report no. TR-47, Massachusetts Institute of Technology, Cambridge, MA, 1985.

....layer. Feed forward ANNs are commonly referred to as perceptrons [26] Feed forward ANNs which had intermediate layers (also called hidden layers) were shown to be inadequate for many larger problems [Minskey, 1969] As a result, back propagation ANNs where introduced to address these shortcomings [3,36,21,27]. In addition to allowing signals to propagate forward, back propagation ANNs allow the outputs to propagate backwards through the network in order to determine intermediate node outputs. This achieved a more robust algorithm for modifying weights and generally decreased the overall training time. ....

Parker, D.B. Learning Logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, Cambridge, MA, 1985.

....solving paradigms but are very well suited to be solved by a neural network 7 as a parallel distributed processing device. One of the most popular and useful types of such a device is the back propagation network. This type of network was first described by Werbos (Werbos,1974) and later by Parker (Parker,1985) but was made popular in a broader range by Rumelhart (Rumelhart et al.,1986) It is a multi layered perceptron using the supervised mode of learning. Let me describe how it operates to give the flavour of how it is capable of solving the kind of problem to which it is applied in this paper. I will ....

D. B. Parker, Learning Logic, Technical Report TR-47, Center for Computational Research in Economics and Management Science, MIT, Cambridge, MA, April 1985.

No context found.

D.B. Parker. Learning logic. Technical Report 47, Center for Computational Research in Economics and Management Science, MIT, April 1985.

No context found.

D.B. Parker. Learning logic. Technical Report 47, Center for Computational Research in Economics and Management Science, MIT, April 1985.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC