Moriarty, D.E. & Miikkulainen, R. "Efficient reinforcement learning through Symbiotic evolution." Machine Learning 22, 1996. pp 11-33  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... individual by testing the performance of the system consisting of this individual and the best individuals from other populations [5] To get a better estimation, we can construct many systems, and determine the fitness of an individual by evaluating the performance of all systems it participates [6]. In our study, we have also proposed two CCEAs for learning of nearest neighbor based multilayer perceptrons (NN MLP) The first one is the individual evolutionary algorithm (IEA) The IEA can construct an NN MLP efficiently if good candidates of the modules are given [8] Compared with other ....

....evaluation) changes in every learning cycle. Third, all populations cannot evolve in parallel because evaluation of individuals in one population requires all other populations being frozen. B. The Symbiotic, Adaptive Neuro Evolution In the symbiotic, adaptive neuro evolution (SANE) algorithm [6], many organizations are constructed in each learning cycle. Therefore, if we consider only one learning cycle, SANE is similar to the society model. The task of SANE is to find neural networks suitable for function approximation. In each organization O i , the task set T i is nothing but the ....

D. E. Moriarty and R. Mikkulainen, "Efficient reinforcement learning through symbiotic evolution," Machine Learning, Vol. 22, pp. 11-33, 1996.),

....and this may be used to reduce or increase the complexity of the network, within predefined limits. 6 5 Pole balancing problem To assess the performance of the method proposed, it was applied to the pole balancing problem. This problem is a well studied benchmark for control methods [17, 18, 8, 19, 20, 21, 22]. The task consists of balancing a pole hinged in the center of a moving cart, by applying a force to the cart exclusively. The pole is only allowed to move in a vertical plane and the cart moves in a one dimensional track. The system is illustrated in Figure 4. The only forces acting on the ....

D. Moriarty. Efficient reinforcement learning through symbiotic evolution. In Machine Learning, volume 22, pages 11--33. 1996. 15

....using the Euler s method (i.e. t 1) t) t) with a time step of = 0.02 second. The system is considered to be out of balance when the pendulum falls beyond 12 degrees from the vertical position or the cart runs beyond Sigma2:4 meters from the center. Previous approaches [2] [22], 33] to tackling the inverted pendulum problem employed a feedforward neural network using h; h; and as inputs, and the output was interpreted as the force applied to the cart. While the trained networks are able to balance the pendulum, four input variables are required to represent the ....

D. E. Moriarty and R. Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

....of Texas at Austin Austin, TX 78712 1188 risto cs.utexas.edu Abstract Natural deduction is essentially a sequential decision task, similar to many game playing tasks. Such a task is well suited to benefit from the techniques of neuro evolution. Symbiotic, Adaptive Neuro Evolution (SANE)(Moriarty and Miikkulainen 1996) has proven successful at evolving networks for such tasks. This paper will show that SANE can be used to evolve a natural deduction system on a neural network. Particularly, it will show that (1) incremental evolution through progressively more challenging problems results in more effective ....

....recombined with other good networks to produce offspring networks. If it does not, it receives a low fitness value and may be removed from the population. However, if neuro evolution is done at the level of partial solutions, the process turns out much more effective. This idea is used in SANE (Moriarty and Miikkulainen 1996). SANE evolves two separate populations, one of nodes and another of network blueprints. The node population focuses on developing specific problem solving functionality for the given task. The network population focuses on combining the nodes effectively. The key to successful evolution is ....

[Article contains additional citation context not shown here]

Moriarty, D.E. and Miikkulainen, R. (1996). Efficient Reinforcement Learning through Symbiotic Evolution. Machine Learning, 22:11-32.

....in high quality networks. In the other population, the space of network specifications is searched for good teams of hidden units that result in high performance composite solutions. SANE has been applied successfully to several sequential decision tasks like the pole balancing problem [62], game playing [61] or robot arm control [63] Another approach to cooperative coevolution is the one exploited by Paredis [72] In his work, a population of solutions and a population of permutations performed on the genotype of the first population coevolve. The motivation underlying this work ....

....input vectors. The last element of the list corresponds to the default component for generating the output when none of the previous domains in the list matches the input vector. This architecture and Hierarchical SANE [64] have many features in common. SANE (Symbiotic, Adaptive Neuro Evolution) [62] is an evolutionary inspired system for the design of neural networks applied to problems in reinforcement learning. In its first implementation, SANE was composed of a unique population of evolving agents, each agent representing one hidden unit of a neural network. The fitness of agents is ....

David E. Moriarty and Risto Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--33, 1996.

....of memory and run time. This section briefly discusses some ongoing research designed to solve Large Scale Markov decision problems. An approach which motivates the comparison between genetic algorithms and dynamic programming, being done in chapter 6, is the symbiotic, adaptive neuroevolution [57]. It belongs to the general class of reinforcement learning methods, which are learning methods based on learning an input output mapping through a process of trial and error designed to maximize a scalar performance index. The Symbiotic, adaptive neuro evolution was designed as a method for ....

: Moriarty D.E., Miikkulainen R. 1996. Efficient Reinforcement Learning Through Symbiotic Evolution. Machine Learning 22:11.

....dissertation is a, mainly empirical, study of the evolution of neural networks. More precisely, the topology of the nodes, the links, as well as the connection weights of a neural network are to be determined with the aid of a genetic algorithm. The novel approach of the genetic algorithm employed [MM94a] is that it encodes one neuron per chromosome. Therefore, the fitness of a chromosome is determined by its degree of cooperation with other chromosomes as they form a network. An extension of this genetic algorithm has been considered, which allows multi layer feed forward, as well as recurrent ....

....rather than modeling natural evolution, we should be cautious. It is not suggested that Genetic Algorithms shall be the successor of every search method, they are not a panacea. 1.1. 3 The Symbiotic Approach In the current dissertation we are examining a genetic algorithm which bears the name SANE [MM94a]. The novelty of this approach is that each chromosome encodes just one neuron. Each chromosome carries part of the whole solution. Hence, the chromosomes whom the phenotypes will constitute the neural network, must reach a level of symbiosis. Moreover, the chromosomes do not converge to what ....

[Article contains additional citation context not shown here]

D. E. Moriarty and R. Miikkulainen. Efficient Reinforcement Learning through Symbiotic Evolution. Technical report AI 94-224, Dept of Computer Sciences, The University of Texas at Austin, September 1994.

....and his neurons are binary valued. It does represent one of the few examples of a population of neurons being evolved, although the neural net structure is very rigidly fixed compared to ours, and only works in simulation. Another group to evolve a population of neurons is Moriarty and Mikkulainen [13] who use symbiotic evolution . Their system seems to rely on neurons cooperating in a very weak fashion essentially they cooperate by ignoring one another. Our system encourages much more active cooperation. Another popular alternative to neural networks GAs is Q learning [19] This is most ....

David E. Moriarty and Risto Mikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, (22), 1996.

....the network size, as this determines the chromosome length. To reduce the number of parameters which must be encoded in each chromosome a method of encoding the weights of a single hidden unit in each chromosome and combining them to form a complete network was incorporated in to the algorithm [5]. Chromosomes of this type are evaluated by selecting several from the population at random and combining them into a complete neural network. A large number of such networks are constructed for each generation of the GA, and the chromosomes are assigned a fitness equal to the average fitness of ....

....proportional to the fan in of individual units, and not to the size of the network. Fewer complete networks are represented in the population, so we are trading off chromosome length for population size. This method is intended to reward chromosomes which cooperate well with other chromosomes [5], but the best networks will not necessarily be discovered by random selection, because a good network will require a complementary set of specialized feature detectors. For this reason, a variation of this algorithm was implemented which divides the population into a set of sub populations, ....

D.E. Moriarty, and R. Miikkulainen, "Efficient reinforcement learning through symbiotic evolution", Machine Learning, vol. 22, pp. 11-33, 1996.

.... (Holland and Reitman 1978) Other extensions to current evolutionary paradigms have been proposed to encourage the emergence of niches and species in a single population (DeJong 1975; Deb and Goldberg 1989; Davidor 1991; Forrest, Javornik, Smith, and Perelson 1993; Giordana, Saitta, and Zini 1994; Moriarty and Miikkulainen 1996) in which individual niches compete for the allocation of trials. The use of multiple interacting subpopulations has also been explored as an alternate mechanism for coevolving niches using the so called island model (Grosso 1985; Cohoon, Hegde, Martin, and Richards 1987; Pettey, Leuze, and ....

Moriarty, D. E. and R. Miikkulainen (1996). Efficient reinforcement learning through symbiotic evolution.

....algorithm perfors 30 times faster (on average) than the Adaptive Heuristic Critic algorithm. In a recent work, Moriarty and Miikkulainen have also presented a reinforcement learning method called SANE (Symbiotic, Adaptive Neuro Evolution) which is 9 to 16 times faster than the AHC algorithm [16]. Table 1 also shows the minimum number of trials needed for learning (Best) the maximum number of trials (Worst) and the number of failures (from the 100 tests) for both neurocontrollers. It should be noted that although on average the AHC algorithm needs fewer attempts for learning to balance ....

D. E. Moriarty and R. Miikkulainen. Efficient reinforcement learning through symbiotic evolution. In Machine Learning, volume 22, Kluwer Academic Publishers, 11-33 (1996).

....strengthening the analogy with biological evolution. I. INTRODUCTION Neuroevolution (NE) the artificial evolution of neural networks using genetic algorithms, has shown great promise in reinforcement learning tasks. NE outperforms standard reinforcement learning methods in many benchmark tasks [6, 10, 11]. Neural networks are a good class of decision making systems to evolve because they are capable of representing solutions to many different kinds of problems, and the mapping from genotype to phenotype is generally efficient. NE is particularly well suited to reinforcement learning tasks because ....

....problem of balancing two poles simultaneously without giving velocity inputs to the network. This problem is a known benchmark in the reinforcement learning literature, which makes it possible to compare NEAT to other methods. Pole balancing has been used in RL and NE research for over 30 years [1, 3, 6, 7, 9, 11, 15, 17 20]. It is also a good surrogate for real problems, in part because pole balancing in fact is a real task, and also because the difficulty can be adjusted. We present the hardest such problem, balancing two pole simultaneously without velocities, in order to show that NEAT performs well on a ....

[Article contains additional citation context not shown here]

D. E. Moriarty and Risto Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

....of time. 2.3 Genetic Algorithm Genetic algorithms (GA) are well suited for music composition because they can explore the solution space in parallel and search for optimal combinations. We use genetic algorithms to evolve a network to produce better melodies. Recently, Moriarty and Miikkulainen [10] developed an efficient neuro evolution system called SANE (Symbiotic, Adaptive Neuro Evolution) SANE is quick and explorative in evolving neural networks. Our system uses SANE to effectively guide the evolution to good solutions. The quality of the melodies produced by the networks is measured ....

Moriarty, D. E. and Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution. Machine Learning 22:11--32.

....neural network. predators face is coordinating the chase so that in the end the prey has nowhere to go, which requires a more long term strategy. 4 The Multi Agent ESP Approach The Enforced Subpopulations Method (The ESP 1 ; 7, 8] is an extension of Symbiotic, Adaptive NeuroEvolution (SANE; [17, 18, 16]) SANE is a method of neuro evolution that evolves a population of neurons instead of complete networks. Neurons are selected from the population to form the hidden layer of a neural network, which is evaluated on the problem. The fitness is then passed back to all the partaking neurons of the ....

Moriarty, D. E., and Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32.

....NE is a promising approach to solving reinforcement learning problems for several reasons. Past studies have shown NE to be faster and more efficient than reinforcement learning methods such as Adaptive Heuristic Critic and Q Learning on single pole balancing and robot arm control (Moriarty 1997; Moriarty and Miikkulainen 1996). Because NE searches for a behavior, it is effective in problems with large state spaces. In addition, memory is easily represented through recurrent connections in neural networks, making the method a natural choice for learning non Markovian tasks (Gomez and Miikkulainen 1999) In traditional ....

....literature, which makes it possible to demonstrate the effectiveness of NEAT compared to others. It is also a good surrogate for real problems, in part because pole balancing in fact is a real task, and also because the difficulty can be adjusted. Earlier comparisons were done with a single pole (Moriarty and Miikkulainen 1996), but this version of the task has become too easy for modern methods. Balancing two poles simultaneously is on the other hand challenging enough for all current methods. Therefore, we demonstrate the advantage of evolving structure through double pole balancing experiments. Two poles are ....

[Article contains additional citation context not shown here]

Moriarty, D. E., and Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32.

....approaches. Populations were evolved for 80 generations as in the first experiment, but after each generation, the population diversity was measured. A diversity metric F can be generated by taking the average Hamming distance between every Evolutionary Computation Volume 5, Number 4 385 David E. Moriarty and Risto Miikkulainen 0 0.1 0.2 0.3 0.4 0.5 0 10 20 30 40 50 60 70 80 Diversity (Phi) Generation SANE Neuron SANE Standard Elite Standard Tournament Figure 7. The population diversity for each simulation. The neuron based approaches maintain very high levels of diversity, whereas the network based approaches converge to a ....

....a more explorative search. 5.2 Lesion Studies While the emergence of specializations is clear from the PCA studies, the function of each and the overall division of labor are not. To better understand the role of each specialization in Evolutionary Computation Volume 5, Number 4 389 David E. Moriarty and Risto Miikkulainen 30 20 10 0 10 20 30 30 20 10 0 10 20 30 Generation 0 30 20 10 0 10 20 30 30 20 10 0 10 20 30 Generation 10 30 20 10 0 10 20 30 30 20 10 0 10 20 30 Generation 20 30 20 10 0 10 20 30 30 20 10 0 10 20 30 Generation 40 30 20 10 0 10 20 30 30 20 10 0 10 20 ....

Moriarty, D. E., & Miikkulainen, R. (1996a). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22, 11--32.

No context found.

Moriarty, D. E., and Miikkulainen, R. Efficient reinforcement learning through symbiotic evolution. In Kaelbling, L. P., editor, Recent Advances in Reinforcement Learning, Dordrecht; Boston: Kluwer, 1996.

No context found.

Moriarty, D. E., & Miikkulainen, R. (1996a). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22, 11--32.

....has proven more powerful in a number of benchmark tasks, especially in continuous domains and in domains where the state is incompletely specified. For example, the SANE neuro evolution method has been shown to be more than twice as fast as Q learning in the pole balancing task (Moriarty, 1997; Moriarty and Miikkulainen, 1996). This paper focuses on further developing techniques for evolutionary reinforcement learning. We introduce EuSANE, a method based on a two level evolution of hidden neurons and network blueprints similar to SANE. However, the blueprint evolution takes place through a eugenic evolution algorithm ....

....established itself as the standard benchmark for reinforcement learning methods. Unlike many other dynamical systems, it is conceptually simple and intuitive to humans, yet a good representative of real world control tasks (Barto et al. 1983; Anderson, 1989; Whitley et al. 1993; Pendrith, 1994; Moriarty and Miikkulainen, 1996). However, it is no longer challenging enough for modern reinforcement learning methods and more difficult variants need to be found. One particularly difficult variant is a cart with two poles of different lengths that have to be balanced simultaneously. The 2 pole problem can be solved with ....

[Article contains additional citation context not shown here]

Moriarty, D. E., and Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution.

....from the organization of phonemes in the words of that language. Thus our model, which will be referred to as the Emergent Syllable Systems model, or ESS, simulates the emergence of a vocabulary of words. Architecture of Vocabulary Evolution The ESS model is based on Symbiotic Evolution (Moriarty and Miikkulainen, 1996). In this method, genetic algorithms evolve a population of partial solutions that combine to yield an optimal solution to the given problem. In the present adaptation of this model a set of words are randomly generated to form vocabularies and the fitness of these vocabularies is evaluated ....

Moriarty, D.E. and Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22, 11--32.

....the game is so difficult that new techniques are probably going to be needed before go programs are as strong as those that play checkers, chess, or Othello. This paper explores the usefulness of neuro evolution as a mechanism for learning to play go. The SANE (Symbiotic, Adaptive Neuro Evolution [7, 8, 9]) algorithm demonstrates that networks that display a general ability in playing go on small boards can be evolved without To appear in Applied Intelligence. y This research was supported in part by NSF under grant #IRI 9504317. Sigma Xi Xi Xi Theta 2 3 Delta 1 Gamma Omega ....

....the many moves played were good and deserve credit for a win, and which were bad and deserve to be blamed for a loss. In go, this problem is severe enough that standard learning techniques such as backpropagation cannot be effectively applied. 5 SANE SANE 1 (Symbiotic Adaptive Neuro Evolution [7, 8, 9]) solves the credit assignment problem by using evolutionary algorithms to search for effective neural networks. Instead of punishing or rewarding individual moves, networks are evaluated, selected, and recombined based on their overall performance in the game. Evolutionary algorithms perform a ....

[Article contains additional citation context not shown here]

David Moriarty and Risto Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

....basic pole balancing problem obsolete. It can now be solved so easily that it provides little or no insight about a system s ability. Neuroevolution (NE) systems (i.e. systems that evolve neural networks using genetic algorithms) for example, often find solutions in the initial random population [Moriarty and Miikkulainen, 1996; Gomez and Miikkulainen, 1997] In response to this need for a new benchmark, the basic pole balancing task has been extended in a variety of ways. Wieland[1991] presented several variations to the standard single pole task that can be grouped into two categories: 1) modifications to the ....

....evolution to almost equal pole lengths. The last two sections contain a discussion of the results and the conclusion. 2 Neuro Evolution Method: Enforced Sub Populations Delta Coding. The Neuroevolution method used is based on Symbiotic, Adaptive Neuro Evolution (SANE; Moriarty, 1997; Moriarty and Miikkulainen, 1996). SANE has been shown to be a powerful reinforcement learning method for tasks with sparse reinforcement. 2.1 SANE SANE differs from other NE systems in that it evolves a population of neurons instead of complete networks (figure 1) These neurons are combined to form hidden layers of ....

[Article contains additional citation context not shown here]

David E. Moriarty and Risto Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

No context found.

Moriarty, D. E., and Miikkulainen, R. (1996a). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32.

.... Dayan, 1992; Kaelbling, Littman, Moore, 1996) over rewards, which lets them acquire mappings from state action pairs onto expected values. Another class of methods search more directly through the space of control policies (Grefenstette, Ramsey, Schultz, 1990; Holland Reitman, 1978; Moriarty Miikkulainen, 1996; Whitley, Dominic, Das, Anderson, 1993; Wilson, 1994) often using evolutionary algorithms to this end. Our approach, to which we now turn, relies on an evolutionary algorithm as the primary mechanism for reinforcement learning, but it also incorporates a technique similar to ....

....Right Ahead Speed Move Left Move Right Stay Center Figure 3 The input and outputs to the neural network for lane selection. The learning system relies on three interrelated modules to determine the weights on the network s links, and thus to acquire robust controllers. The first component is SANE (Moriarty Miikkulainen, 1996; Moriarty, 1997) which carries out genetic search through the space of feedforward networks, given a network architecture, by operating at two distinct levels. At one level, the module retains a population of complete controllers, each defined as a collection of hidden layer neurons. The ....

Moriarty, D. E., & Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution.

....methods such as backpropagation. The evolutionary framework frees the implementor from generating training examples and provides a highly adaptive mechanism for dynamic environments. Recent work has shown evolved neuro controllers effective in several unstable, dynamic control tasks [3] [6], 8] 13] The bane of the evolutionary methods, however, has been the large number of fitness evaluations that must be performed to achieve a high level of performance. Recently, we have developed a more efficient evolutionary approach called SANE (Symbiotic, Adaptive NeuroEvolution) 6] which ....

....[3] 6] 8] 13] The bane of the evolutionary methods, however, has been the large number of fitness evaluations that must be performed to achieve a high level of performance. Recently, we have developed a more efficient evolutionary approach called SANE (Symbiotic, Adaptive NeuroEvolution) [6], which explicitly decomposes the evolutionary search for a complete solution into several parallel searches for partial solutions. In most approaches to neuroevolution, each individual represents a complete neural network that is evaluated independently of other networks in the population [2] ....

[Article contains additional citation context not shown here]

David E. Moriarty and Risto Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

....that is similar in spirit to temporal difference methods. Figure 3 illustrates the interaction of the different learning methods, which are described in the next three sections. Reinforcement Learning using SANE The backbone of the learning system is the SANE reinforcement learning method (Moriarty Miikkulainen, 1996; Moriarty, 1997) This section gives a brief outline of SANE and its advantages; the aforementioned references provide more detailed information. SANE (Symbiotic, Adaptive Neuro Evolution) was designed as an efficient method for forming decision strategies in domains where it is not possible to ....

Moriarty, D. E., & Miikkulainen, R. (1996). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22, 11--32.

No context found.

Moriarty, D. E., and Miikkulainen, R. (1996a). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32.

No context found.

David E. Moriarty and Risto Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

No context found.

Moriarty, D. E., and Miikkulainen, R. (1994a). Efficient reinforcement learning through symbiotic evolution. Technical Report AI94-224, Department of Computer Sciences, The University of Texas at Austin.

....basic pole balancing problem obsolete. It can now be solved so easily that it provides little or no insight about a system s ability. Neuroevolution (NE) systems (i.e. systems that evolve neural networks using genetic algorithms) for example, often find solutions in the initial random population [5, 6]. In response to this need for a new benchmark, a variety of ways to extend the basic pole balancing task have been suggested. Wieland [7] presented a series of increasingly difficult variations on the standard pole balancing task This research was supported in part by National Science ....

....Results are the average of 50 simulations. derivatives. This is a realistic version of the problem since only the positions can be observed easily in the real world. The SANE approach has proven faster and more efficient than other reinforcement learning methods in the basic pole balancing task [6] and ESP has been shown to solve the double pole balancing problem very efficiently [5] The two dimensional problem examined here was found to be more difficult to evolve than the double pole problem. There are several factors that make this problem more challenging for NE systems: 1) The ....

[Article contains additional citation context not shown here]

D. E. Moriarty and R. Miikkulainen. Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32, 1996.

No context found.

Moriarty, D. E., and Miikkulainen, R. (1996a). Efficient reinforcement learning through symbiotic evolution. Machine Learning, 22:11--32.

No context found.

Moriarty, D.E. & Miikkulainen, R. "Efficient reinforcement learning through Symbiotic evolution." Machine Learning 22, 1996. pp 11-33

No context found.

D. E. Moriarty and R. Mikkulainen, "Efficient reinforcement learning through symbiotic evolution," Machine Learning, Vol. 22, pp. 11-33, 1996.),

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC