#Heskes, T.M., B. Kappen, On-line learning processes in artificial neural networks, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993). 15  Home/Search   Document Details and Download   Summary   Related Articles   Check

....have been developed so far that meet some of the criteria above and that have influenced the development of EFuNNs. These are methods and systems for: adaptive learning [4,5,7,8,14,30,46,47,48] incremental learning [6,7,8,9,19,53,58,61,71] lifelong learning [69,35,36,82] on line learning [17,21,22,28,31,35,36,42,44,61,66,67,69]; constructivist structural learning [15,19,11,14,9] that is supported by biological facts [14,62,73,77,82] selectivist structural learning [26,29,49,56,59,64,50,32] hybrid constructivist selectivist structural learning December,2001 3 [52,66,70,31] knowledge based learning neural networks ....

Heskes, T.M., B. Kappen, "On-line learning processes in artificial neural networks", in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993).

....has learned and what it knows about the problem it is trained to solve; make decisions for a further improvement. Some of the above seven issues have already been addressed in different connectionist systems. Such systems can successfully perform incremental learning [7 15] on line learning [9,13,20]; can deal with rules [3,5,17,32,34,40,49,50,55] The latter class of neural networks (NN) are also called knowledge based neural networks (KBNN) On line learning is concerned with learning data as the system operates (usually in a real time) and data might exist only for a short time. NN models ....

....latter class of neural networks (NN) are also called knowledge based neural networks (KBNN) On line learning is concerned with learning data as the system operates (usually in a real time) and data might exist only for a short time. NN models for on line learning are introduced and studied in [1,9,13,20,34,55]. Several investigations proved that the most popular neural network models and algorithms are not suitable for adaptive, on line learning, that include: multi layer perceptrons trained with the backpropagation algorithm, radial basis function networks [13] self organising maps (SOMs) 35,36] ....

T.M. Heskes, B. Kappen, On-line learning processes in artificial neural networks, in: Math. Foundations of Neural Networks, Elsevier, Amsterdam, 1993, pp. 199-233. 20

....and systems have been developed so far that meet some of the criteria above and that have influenced the development of EFuNNs. These are methods and systems for: adaptive [6,7,8,9,19,53,58,61,71] learning [4,5,7,8,14,30,46,47,48] incremental lifelong learning [69,35,36,82] on line [17,21,22,28,31,35,36,42,44,61,66,67,69]; constructivist structural [ 15,19,11,14,9] that is supported by biological facts [ 14,62,73,77, 82] selectivist structural learning [26,29,49,56,59,64,50,32] hybrid constructivist selectivist structural learning 2 [52,66,70,31] knowledge based learning neural networks (KBNN) ....

Heskes, T.M., B. Kappen, "On-line learning processes in artificial neural networks", in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993).

....patterns. However, this method does not guarantee what kind of unknown patterns can be classified in the same category. Several neural network theories and methods for adaptive learning and for dynamic modification of neural network structures have been introduced so far: incremental learning [1]; growing neural networks [2, 6] pruning neural networks [3, 4, 5] A problem of these networks is that their theoretical properties are not related directly to the concrete design of the general pattern. In fact, these properties say that the networks have excellent approximation properties, but ....

T.M. Heskes and B. Kappen, "On-line learning processes in artificial neural networks, " in Math. foundations of neural networks. Amsterdam: Elsevier, 1993. pp.199-233.

....Batch learning will discover the minimum of whatever basin the weights are initially placed. In stochastic learning, the noise present in the updates can result in the weights jumping into the basin of another, possibly deeper, local minimum. This has been demonstrated in certain simplified cases [15, 30]. Stochastic learning is also useful when the function being modeled is changing over time, a quite common scenario in industrial applications where the data distribution changes gradually over time (e.g. due to wear and tear of the machines) If the learning machine does not detect and follow ....

T.M. Heskes and B. Kappen. On-line learning processes in artificial neural networks. In J. G. Tayler, editor, Mathematical Approaches to Neural Networks, volume 51, pages 199--233. Elsevier, Amsterdam, 1993.

....manipulation as discussed below. Adaptive learning is aimed at solving the well known stability plasticity dilemma [4, 3, 5] Several methods for adaptive learning are related to the work presented here, namely incremental learning [12, 11, 40] lifelong learning [24] and on line learning [9, 48, 47, 15]. Incremental learning is the ability of a NN to learn new data without destroying (or at least fully destroying) the learned patterns from old data, and without a need to be trained on the entirety of the old and new data. On line learning is concerned with learning data as the system operates ....

T.M. Heskes and B. Kappen. On-line learning processes in artificial neural networks. In Math. foundation of neural networks. Elsevier, Amsterdam, 1993.

....it knows about the problem it is trained to solve; to make decision about its own improvement. Several NN theories, models and methods for adaptive learning and for dynamical modification of NN structures have been introduced so far: incremental learning [1] lifelong learning; on line learning [3]; growing NN [2] pruning NN [10,8,4] etc. A framework called ECOS (Evolving COnnectionist System) that addresses all the issues above is introduced in the paper along with a method of training called eco training. They are illustrated on bench mark data and real world data (Iris classification ....

Heskes, T.M., Kappen, B. On-line learning processes in artificial neural networks, in: Math. foundations of neural networks, Elsevier, Amsterdam, (1993)199-233

....t must not be perpendicular to the gradient. 0 w 6 1 w o = 0;99 R Figure 4: Learning rate adaptation (4) 9) for w 7 p jwj with j o = 0;1 and i = 1;3 The algorithms in this article are made for batch learning, i.e. the minimisation of a specific cost function as opposed to online learning (Heskes and Kappen 1993; Amari 1993; Barkai et al. 1995) where successive steps aim at minimising different cost functions (each given by a single pattern) Although the methods proposed in Section 3 cannot be expected to converge always in the case of online learning, experiments have shown that they work quite well ....

Heskes, T. M. and H. J. Kappen (1993). On-line learning processes in artificial neural networks.

.... training set, the more attractive on line learning rules if compared with (more advanced) batch mode learning rules (see e.g. 1] Pattern by pattern presentation introduces stochasticity in the learning process that can be studied using techniques from theory on stochastic processes (see e.g. [2, 3]) These studies have been mainly focussed on memory less on line learning or randomized learning , that is, at each learning step an example is drawn at random from the training set, without any reference to previous presentations. In this paper we will extend the theory to correlated ....

.... point out the most important results (see [7, 4, 8] for more details) 1 Real World Computing Program 2 Dutch Foundation for Neural Networks 2 Tom Heskes and Wim Wiegerinck 2 Theoretical framework Our analysis combines the time averaging procedure proposed in [2] with Van Kampen s expansion [3]. After M iterations of the learning step (1) we have w(n M) Gamma w(n) j M Gamma1 X m=0 hf(w(n m) x(n m) i ; where h Deltai and w refer to averages over both the pattern dynamics and the weight dynamics. For small learning parameters the weight dynamics is a factor j slower than ....

[Article contains additional citation context not shown here]

T. Heskes and B. Kappen. On-line learning processes in artificial neural networks. In J. Taylor, editor, Mathematical Foundations of Neural Networks, pages 199--233. Elsevier, Amsterdam, 1993.

....of patterns x, on line learning as described by (1) is a stochastic process. The probability to be in a certain network state w can be shown to obey a master equation. In recent years, theoretical studies of this master equation have provided a better understanding of on line learning processes [1, 2, 3, 4, 5, 6]. When the last weight change is added to the learning rule (1) the weight change takes the form Deltaw(n) j f(w(n) x) ff Deltaw(n Gamma 1) 2) with ff the so called momentum parameter. Equation (2) describes on line learning with momentum term. The incorporation of this momentum term ....

....we will study the two dimensional system (3) for small values of ffl and finite values of fl, i.e. in the limits j 0 and ff 1 with a constant ratio fl = j= 1 Gamma ff) 2 . The master equation (4) can be approximated for small parameters ffl using Van Kampen s expansion. Basically (see [12, 5, 6] for a more detailed description of Van Kampen s expansion) this expansion is based on the assumption that the stochastic process (3) can be viewed as a deterministic trajectory with (small) superimposed fluctuations of order p ffl. Starting from the Ansatze w = OE p ffl and q = p ....

[Article contains additional citation context not shown here]

T. Heskes and B. Kappen. On-line learning processes in artificial neural networks. In J. Taylor, editor, Mathematical Foundations of Neural Networks, pages 199--233. Elsevier, Amsterdam, 1993.

....(unsupervised) Kohonen learning rule [10] and the (supervised) backpropagation learning rule [11] see sections 3.3 and 3.4) On line learning described by (1) is a Markov process. The probability p i (w) for the system to be in state w after i learning iterations obeys the random walk equation [12, 2, 13] p i 1 (w) Z dw 0 T (wjw 0 ) p i (w 0 ) 2) with transition probability T (wjw 0 ) to go from an old state w 0 to a new one w given by T (wjw 0 ) Z dx ae(x) ffi(w Gamma w 0 Gamma j f(w 0 ; x) 3) with ae(x) the probability density function of training patterns. We ....

....on the transition matrix T (wjw 0 ) are stated that justify a full use of the FokkerPlanck approximation (7) see also section 2. 3) These conditions do not hold for the transition probability (3) For a further explanation we refer to the standard text books [15, 16] or the book chapter [13]. 2.2 Hansen s Fokker Planck equation Hansen et al. 3] arrive at a slightly different Fokker Planck equation through a quite different route. They average the dynamics of the weights (1) over a large number 1 n 1=j of learning steps. Neglecting higher order terms and assuming independence ....

[Article contains additional citation context not shown here]

T. Heskes and B. Kappen. On-line learning processes in artificial neural networks. In J. Taylor, editor, Mathematical Foundations of Neural Networks, pages 199--233. Elsevier, Amsterdam, 1993.

....(1) with w the weight vector, which includes the strength of all synapses and thresholds, j the learning parameter, and f( the backpropagation learning rule. In the following we will use one dimensional notation for simplicity. The learning process (1) can be described by the master equation [4, 1, 5] t P (w; t) P (w; t) Z dw 0 T (wjw 0 ) P (w 0 ; t) 2) with the transition probability to go from an old state w 0 to a new one w, T (wjw 0 ) 1 p p X =1 ffi(w Gamma w 0 Gamma j f(w 0 ; x ) With a smart choice of the time intervals between subsequent ....

.... the transition probability to go from an old state w 0 to a new one w, T (wjw 0 ) 1 p p X =1 ffi(w Gamma w 0 Gamma j f(w 0 ; x ) With a smart choice of the time intervals between subsequent adaptations, the master equation (2) exactly describes the learning process (1) [5]. In general, this master equation cannot be 1 Proceedings of the European Symposium on Artificial Neural Networks 94, pages 223 228 solved analytically. An option is to look for approximations valid for small learning parameters j. The first step in most approximation schemes is to write ....

[Article contains additional citation context not shown here]

T. Heskes and B. Kappen. On-line learning processes in artificial neural networks. In J. Taylor, editor, Mathematical Foundations of Neural Networks, pages 199--233. Elsevier, Amsterdam, 1993.

No context found.

#Heskes, T.M., B. Kappen, On-line learning processes in artificial neural networks, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993). 15

No context found.

Heskes, T.M., B. Kappen, "On-line learning processes in artificial neural networks", in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993).

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC