D.J. Janson and J.F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, 1993.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....to adopt in a changing environment. In the literature, research into EANN has been taking one of three approaches; evolving the weights of the network, evolving the architecture, or evolving both simultaneously. The EANN approach uses either binary representation to evolve the weight matrix [12, 13] or real [7, 8, 9, 18, 19, 23] There is not an obvious advantage of binary encoding in EANN over the real. However, with real encoding, there are more advantages including compact and natural representation. The key problem (other than being trapped in a local minimum) with BP and other ....

D.J. Janson and J.F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, 1993.

....to adopt in a changing environment. In the literature, research into EANN has been taking one of three approaches; evolving the weights of the network, evolving the architecture, or evolving both simultaneously. The EANN approach uses either binary representation to evolve the weight matrix [10, 11] or real [6, 7, 16, 19] There is not an obvious advantage of binary encoding in EANN over the real. However, with real encoding, there are more advantages including compact and natural representation. The key problem (other than being trapped in a local minimum) with BP and other traditional ....

D.J. Janson and J.F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, 1993.

.... of Fogel et al. 19] Yao and Liu [87] and Montana and Davis [57] There are also attempts to evolve recurrent networks, e.g. Angeline et al. 3] and McDonnell and Waagen [51] Applying evolutionary search to more complex types of neural networks (high order networks, for example) can be found in [33 35,85], and a good review of evolving neural networks is provided by [86] Back et al. 4] and Fogel [17] provided an introduction to various evolutionary search algorithms. Generally, a population of candidate solutions, ranked by their performance, are maintained and updated iteratively by ....

D. J. Janson and J. F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, 1993.

....termed chromosomes. Some of the early work in evolving ANN connection weights followed this approach Fig. 3. a) An ANN with connection weights shown. b) A binary representation of the weights, assuming that each weight is represented by four bits. 24] 26] 28] 37] 38] 41] 52] [53]. In such a representation scheme, each connection weight is represented by a number of bits with certain length. An ANN is encoded by concatenation of all the connection weights of the network in the chromosome. A heuristic concerning the order of the concatenation is to put connection weights ....

....fitness) function and thus is particularly appealing when this information is unavailable or very costly to obtain or estimate. For example, the evolutionary approach has been used to train recurrent ANN s [41] 60] 65] 100] 102] 103] 106] 117] 126] 128] higher order ANN s [52] [53], and fuzzy ANN s [76] 77] 129] 130] Moreover, the same EA can be used to train many different networks regardless of whether they are feedforward, recurrent, or higher order ANN s. The general applicability of the evolutionary approach saves a lot of human efforts in developing different ....

D. J. Janson and J. F. Frenzel, "Training product unit neural networks with genetic algorithms," IEEE Expert, vol. 8, pp. 26--33, May 1993.

....can be reduced, and the error in approximation decreased. Several neural network architectures have been developed to incorporate higher order terms. These architectures include second order NNs [12] higher order NNs [4] 13] sigma pi NNs [6] functional link NNs [8] and product unit NNs [1] [10], 11] This paper concentrates on the training of product unit neural networks (PUNN) Gradient descent (GD) is possibly the most popular optimization algorithm to train multilayer NNs. While GD has shown to be successful in training SUNNs, GD fails to train PUNNs in general. PUs introduce more ....

....III. A summary of the global optimization algorithms used in this study is given in section III. Results are presented and discussed in section IV. II. Product Unit Training Rule Product unit neural networks were introduced by Durbin and Rumelhart [1] and further explored by Janson and Frenzel [10] and Leerink et al. [11] Instead of using the usual summation units where the net input signal is computed as net y j ;p = I X i=1 z i;p v ji (1) product units are used, where net y j ;p = I Y i=1 z v ji i;p (2) In equations (1) and (2) net y j ;p is the net input signal to unit y j ....

[Article contains additional citation context not shown here]

DJ Janson and JF Frenzel, Training Product Unit Neural Networks with Genetic Algorithms, IEEE Expert Magazine, pp 26-33, October 1993.

....and training time, as well as smaller network architectures. Several neural network architectures have been developed to incorporate higher order terms. These architectures include second order NNs [16] higher order NNs [6, 17] sigma pi NNs [8] functional link NNs [12] and product unit NNs [2, 14, 15]. This paper concentrates on product unit neural networks (PUNN) investigating optimization algorithms to train such networks. Gradient descent (GD) is possibly the most popular optimization algorithm to train multi layer NNs. While gradient descent (GD) has shown to be successful in training ....

....section 3. Global optimization algorithms appropriate for training PUNNs are discussed in section 4. Results are presented and discussed in section 5. 2 Product Unit Training Rule Product unit neural networks were introduced by Durbin and Rumelhart [2] and further explored by Janson and Frenzel [14] and Leerink et al. [15] In PUNNs the net input signal to non input units is computed as a weighted product net y j ;p = I Y i=1 z v ji i;p (1) The usual summation units, where the net input signal is computed as the weighted sum net y j ;p = I X i=1 z i;p v ji (2) are replaced with ....

[Article contains additional citation context not shown here]

DJ Janson and JF Frenzel, Training Product Unit Neural Networks with Genetic Algorithms, IEEE Expert Magazine, pp 26-33, October 1993.

.... 11] sigma pi NNs [4] and functional link NNs [7] This paper concentrates on another alternative, referred to as product unit neural networks (PUNN) Description of a Typical Product Unit Neural Network PUNNs were introduced by Durbin and Rumelhart [2] and further explored by Jansen and Frenzel [8] and Leerink et al. [9] In PUNNs the hidden layer summation units are replaced by product units (PU) to compute the weighted product of inputs: net y j = I Y i=1 z v ji i (1) instead of net y j = I X i=1 z i v ji (2) where net y j is the net input to hidden unit y j , z i is an input ....

....Traditional networks are usually trained using GD. GD optimization works best when the solution space is relatively smooth, with a few local minima or plateaus. Unfortunately, the solution space for product networks can be extremely convoluted, with numerous local minima that trap GD [2, 8, 9]. Leerink et al. have concluded that the parity 6 problem could not be trained by using product units and standard backpropagation [9] They have found two main reasons for this: a) weight initialization and (b) the presence of local minima. In the backpropagation procedure the rst step is to ....

[Article contains additional citation context not shown here]

DJ Janson and JF Frenzel, Training Product Unit neural networks with Genetic Algorithms, IEEE Expert Magazine, pp.26-33, October 1993.

....multiple error measures. A method of error smoothing is also introduced as an attempt to solve the current stopping problem associated with ES (and GA) trained NNs. The use of Evolutionary and Genetic approaches to Neural Network training has received increasing attention in recent years [4, 5, 6, 7, 9, 12, 14]. Indeed the ability of these approaches to facilitate NN training beyond the Euclidean objective was highlighted by Porto et al. [9] but apparently taken no further. Other limited approaches to multi objective training do appear in the literature, but in the form of simultaneously choosing the ....

....sets (see Table 1) These time series were chosen due to the breadth and diverse nature of the time series generation processes. C. Learning rules and stopping criteria The core ES learning routine is that presented in Eq. 3, the population selection routine will follow the methodology of [6]. 1) ES population Selection Algorithm (a) Initiate 100 Networks with a randomised weight vectors (between 0.1 and 0.1) b) Perturb weights by ES. c) Rank by fitness. d) Select base networks for next generation. e) Go to (b) or stop if stopping criteria met. The perturbation in (b) is ....

Janson, D.J. and Frenzel, J.F., "Training Product Unit Neural Networks with Genetic Algorithms", IEEE Expert, October, pp26-33,1993.

.... [130, 257] Fuzzy Sets and Systems (Netherlands) 345] 11 12 Genetic algorithms and neural networks IEEE Aerospace and Electronic Systems Magazine, 186] IEEE Computer Society Technical Committee on Microprogramming and Microarchitecture, 890] IEEE Control Systems, 185] IEEE Expert, [276, 282, 355, 698, 943] IEEE Expert (USA) 489] IEEE Potentials, 699] IEEE Trans. Ind. Appl. USA) 564] IEEE Transaction on Neural Networks, 576] IEEE Transactions on Circuits and Systems I, Fundamental Theory and Applications, 647] IEEE Transactions on Evolutionary Computing, 504] IEEE Transactions ....

....Fogel, Lawrence J. 276, 681, 685, 687, 691] Foo, Shou King, 170] Forst, C. V. 248] Fortuna, L. 639, 640] Foy, Mark, 693] Foy, M. 694] Franco, Aurali B. 888] Fredriksson, Kimmo, 496, 517] Freedman, M. T. 487] Freisleben, Bernd, 695] French, I. G. 171] Frenzel, James F. [696, 697, 698, 699] Friedrich, Ch. M. 497] Fu, Chi Yung, 240] Fujii, T. 498] Fujimoto, Yoshiji, 561] Fujita, S. 36] Fukuda, Toshio, 147, 249, 257, 294, 310, 552, 569, 700, 701, 702, 703, 704, 705, 706, 707, 708] Fukumi, M. 336, 444, 466] Fukumi, Minoru, 709, 710] Fullmer, Brad, 838] Funabiki, ....

[Article contains additional citation context not shown here]

David J. Janson and James F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, October 1993. ga:Frenzel93a.

....Fogel, Lawrence J. 276, 681, 685, 687, 691] Foo, Shou King, 170] Forst, C. V. 248] Fortuna, L. 639, 640] Foy, Mark, 693] Foy, M. 694] Franco, Aurali B. 888] Fredriksson, Kimmo, 496, 517] Freedman, M. T. 487] Freisleben, Bernd, 695] French, I. G. 171] Frenzel, James F. [696, 697, 698, 699] Friedrich, Ch. M. 497] Fu, Chi Yung, 240] Fujii, T. 498] Fujimoto, Yoshiji, 561] Fujita, S. 36] Fukuda, Toshio, 147, 249, 257, 294, 310, 552, 569, 700, 701, 702, 703, 704, 705, 706, 707, 708] Fukumi, M. 336, 444, 466] Fukumi, Minoru, 709, 710] Fullmer, Brad, 838] Funabiki, ....

....Witold, 166] Jackson, Bernie, 925] Jacob, C. 113, 128] Jacob, P. J. 537] Jagielska, I. 514] Jain, L. C. 186, 225, 286, 476] Jain, Sandeep D. 48] Jain, Sandeep, 470] Jakobi, Nick, 353] James, J. 184] James Roxby, Philip B. 210] Jang, Dongsig, 387] Janson, David J. [696, 697, 698] Jarmulak, J. 515] Jefferson, M. F. 583] Jenkins, W. M. 354] Jeon, Jeong Yul, 183] Jeong, Il Kwon, 187] Jerabek, V. 259] Jesung, Ahn, 114] Jia, P. F. 277] Jiang, J. 516] Jockusch, S. R. 821] Jockusch, Stefan, 49] Johns, A. T. 562, 566] Johnson, John D. 26] ....

[Article contains additional citation context not shown here]

David J. Janson and James F. Frenzel. Training product unit neural networks with genetic algorithms. In Firooz A. Sadjadi, editor, Adaptive and Learning Systems, volume SPIE-1706, pages 32--38, Orlando, FL, 20. -21. April 1992. The International Society for Optical Engineering. y(EI 147080/93 CCA 34484/94) ga:Frenzel92b.

....[59] 11 12 Genetic algorithms of 1993 Future Generation Computer Systems, 291] Geophysical Research Letters, 656, 885] Guangxue Xuebao, 363] Helsingin Sanomat, 804, 805, 1031, 1032] IEE Proceedings E Comput. Digit. Tech. 175] IEEE Control Systems Magazine, 667] IEEE Expert, [78, 318, 389, 390, 1034] IEEE Potentials, 319] IEEE Transactions of Power Delivery, 357] IEEE Transactions on Circuits and Systems I, Fundamental Theory and Applications, 177] IEEE Transactions on Circuits and Systems for Video Technology, 631] IEEE Transactions on Computer Aided Design of Integrated ....

....Forrest, Stephanie, 306, 307, 308, 309, 310, 311, 312, 313] Fortuna, L. 156, 157, 158] Fox, B. L. 314] Foy, M. 315] Franco, Aurali B. 854] Franich, R. E. H. 116, 117] Frederick, W. G. 316] Freeman, James, 500] Freeman, L. C. 609] Freisleben, Bernd, 317] Frenzel, James F. [318, 319] Friedman, Michael, 674] Fuentes, Olac, 599] Fujii, S. 320] Fujikawa, Hideji, 884] Fujimoto, Yoshiji, 321] Fujita, Kikuo, 322] Fukuda, T. 533, 534] Fukuda, Toshio, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343] Fukumi, ....

[Article contains additional citation context not shown here]

David J. Janson and James F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, October 1993. ga:Frenzel93a.

....Fonseca, Carlos M. 193] Fontain, Eric, 212, 213] Fontana, Walter, 214] Forrest, Stephanie, 215, 216, 217, 218, 564] Fortuna, L. 117] Foy, Mark D. 264] Frank, P. 120] Frankhauser, Pierre, 219] Frazer, J. H. 345] Freeman, L. M. 375] Freeman, Ray, 565] Frenzel, James F. [220, 221] Freund, Harald, 222] Freyer, Stephan, 223] Frieder, Ophir, 635] Fukuda, Toshio, 224, 225, 226, 227, 228, 229, 230, 231] Fullmer, Brad, 496] Furst, M. 532] Furuhashi, Takeshi, 674] Furuya, Tatsumi, 719] Gall, A. Le, 182] Gallagher, John C. 85] Galletly, J. E. 239] Gammack, ....

....[670] Inaba, Makoto, 224] Inayoshi, H. 398] Ingber, Lester, 339] Ireson, N. S. 198] Ishihara, Toshihisa, 523] Ismail, H. S. 340] Iwamoto, Takashi, 341] Jackson, Bernie, 648] Jakob, Willfried, 266] Janakiraman, Janani, 606] Janikow, Cezary Z. 352, 353] Janson, David J. [220, 221] Jefferson, David R. 354, 567] Jenkins, W. M. 355, 356] Jensen, Eric Dean, 357, 614] Jensen, J. B. 527] Jeon, Hong Tae, 283] Ji, Zhiming, 435] Jiang, Minga, 359] Jin, Lin Ming, 600, 601, 603] Jockusch, S. R. 475] Johnsen, Sonke, 379] Jones, A. H. 37] Jones, Donald R. ....

[Article contains additional citation context not shown here]

David J. Janson and James F. Frenzel. Training product unit neural networks with genetic algorithms. In Firooz A. Sadjadi, editor, Adaptive and Learning Systems, volume SPIE-1706, pages 32--38, Orlando, FL, 20. -21. April 1992. The International Society for Optical Engineering. y(EI 147080/93 CCA 34484/94) ga:Frenzel92b.

....or even continuous since EAs do not depend on gradient information. Because EAs can treat large, complex, nondifferentiable and multimodal spaces, which are the typical case in the real world, considerable research and application has been conducted on the evolution of connection weights [24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112]. The evolutionary approach to weight training in ANNs consists of two major phases. The first phase is to decide the representation of connection weights, i.e. whether in the form of binary strings or not. The second one is the evolutionary process simulated by an EA, in which search operators ....

....Figure 2: A typical cycle of the evolution of connection weights. 2.1 Binary Representation The canonical GA [13, 14] has always used binary strings to encode alternative solutions, often termed chromosomes. Some of the early work in evolving ANN connection weights followed this approach [24, 26, 28, 37, 38, 41, 52, 53]. In such a representation scheme, each connection weight is represented by a number of bits with certain length. An ANN is encoded by concatenation of all the connection weights of the network in the chromosome. A heuristic concerning the order of the concatenation is to put connection weights to ....

[Article contains additional citation context not shown here]

D. J. Janson and J. F. Frenzel, "Training product unit neural networks with genetic algorithms, " IEEE Expert, vol. 8, no. 5, pp. 26--33, 1993.

....dynamic scheduling of computational resources and the identification of promising ANNs based on partial TSSE versus time information. PLAN has a key difference on how reinforcement in learning is done with respect to Janson and Frenzel s genetics based machine learning system for designing ANNs [19]. Janson and Frenzel computed the fitness of a partially trained ANN as a function of its sum of squared errors (SSE) and used the fitness to rank all the ANNs. Since the instantaneous SSE is not monotonic with respect to training time, and many other factors (such as the rate of change of SSE ....

D. J. Janson and J. F. Frenzel. Training product unit neural networks with genetic algorithms. Expert: Special Issue on Intelligent Systems and their Applications, 8:26--33, October 1993.

.... neural networks [1] In addition to the obvious biological appeal of the idea, evolutionary techniques such as Genetic Algorithms have an advantage over more popular supervised learning techniques in that they can be used to train neural networks with unrestricted architectures and neuron types [2,3], and their blind search characteristic means that networks can be evolved to perform unsupervised learning tasks where no immediate error information can be provided about the network output. Performance depends on the existence of satisfactory networks within the search space of allowed ....

Janson D.J., Frenzel J.F.; 'Training Product Unit Neural Networks with Genetic Algorithms', IEEE Expert. 26-33, Oct 1993

.... Genetic algorithms (GAs) have been demonstrated to be a promising search and optimization technique [1] It has been successfully applied to system identification [2, 3, 4, 5] and a wide range of applications including design [6] scheduling [7] routing [8] control [9, 10] and others [11, 12, 13]. One of the main obstacles in applying GAs to complex problems has often been the high computational cost due to their slow convergence rate. The convergence rate of a GA is typically slower than that of local search techniques (e.g. steepest descent) because it does not use much local ....

D. J. Janson and J. F. Frenzel, "Training product unit neural networks with genetic algorithms," IEEE Expert, vol. 8, no. 5, pp. 26--33, 1993.

.... Genetic algorithms (GAs) have been demonstrated to be a promising search and optimization technique [1] It has been successfully applied to system identification [2, 3, 4, 5] and a wide range of applications including design [6] scheduling [7] routing [8] control [9, 10] and others [11, 12, 13]. One of the main obstacles in applying GAs to complex problems has often been the high computational cost due to their slow convergence rate. The convergence rate of a GA is typically slower than that of local search techniques (e.g. steepest descent) because it does not use much local ....

D. J. Janson and J. F. Frenzel, "Training product unit neural networks with genetic algorithms," IEEE Expert, vol. 8, no. 5, pp. 26--33, 1993.

....since GAs do not depend on gradient information in search. Because GAs are good at dealing with large, complex, nondifferentiable and multimodal spaces which are the typical space defined by an error function or fitness function, a lot of work has been done on the evolution of connection weights [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64]. The evolutionary approach to weight training in EANNs consists of two major stages. The first stage is to decide the genotype representation of connection weights, i.e. whether in the form of binary strings or not. The second one is the evolution itself simulated by a GA or other evolutionary ....

....weights. Reprinted with permission from Ref. 1. X. Yao: Evolutionary Artificial Neural Networks 9 2. 1 Binary Representation Since the binary representation has been shown to be beneficial in GA s search [4, 5] one way to represent connection weights is to encode them in binary strings [29, 30, 32, 41, 42, 45, 56, 57]. In such a representation scheme, each connection weight is represented by a number of binary bits with certain length. An EANN is represented by concatenation of all the connection weights in the network. A heuristic concerning the order of the concatenation is to put connection weights to the ....

[Article contains additional citation context not shown here]

D. J. Janson and J. F. Frenzel. Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26--33, 1993.

....error signal. Thus, to apply evolutionary algorithms seems to be advantageous at least to problems where gradient information is difficult to obtain, e.g. to recurrent networks, to networks with non differentiable transfer functions or non differentiable optimality criteria, to product neurons [34] or threshold neurons. One drawback of genetic algorithms is that they seem to have difficulties to fine tune the parameters [39] And there is the problem of competing conventions discussed in Section 6. So far, evolutionary algorithms in this area do not seem to be competitive with improved ....

....the parameters [39] And there is the problem of competing conventions discussed in Section 6. So far, evolutionary algorithms in this area do not seem to be competitive with improved gradient decent methods like quickprop or cascade correlation, see [39, 65] For some work in that area see e.g. [13, 34, 52, 69, 76, 77, 78, 81]. In [15, 41] recurrent nets are trained. 2.1 Representation The straightforward genotype representation is simply a concatenation of all the network s weights in a string. Since the standard single point crossover operator is more likely to disrupt genes that are far apart on the chromosome ....

[Article contains additional citation context not shown here]

D. J. Janson and J. F. Frenzel. Training product unit neural networks with genetic algorithms, 1993.

....viable and they survive to reproduction. The population evolves over time producing solutions according to setted fitness function. 9, 17] The genetic algorithm can be applied to a wide range of optimization and learning problems, including learning the topology and the weights of neural networks [11], optimization of diesel engine cam shaft[1] shape and topology optimization [16] electrical circuit optimization[18] power system parameter optimization[7] and torque and efficiency optimization of an inductive motor[15] 1.3 General description of the sales support system. The sales support ....

David J. Janson and James F. Frenzel. Training product unit neural networks with genetic algorithm. IEEE Expert, 8(Octpber 1993):26--33, 1993.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC