Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples -- these are among the basic building blocks making up the architecture -- yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.

In many classifier systems, the classifier strength parameter serves as a predictor of future payoff and as the classifier's fitness for the genetic algorithm. We investigate a classifier system, XCS, in which each classifier maintains a prediction of expected payoff, but the classifier's fitness is given by a measure of the prediction's accuracy. The system executes the genetic algorithm in niches defined by the match sets, instead of panmictically. These aspects of XCS result in its population tending to form a complete and accurate mapping X x A => P from inputs and actions to payoff predictions. Further, XCS tends to evolve classifiers that are maximally general subject to an accuracy criterion. Besides introducing a new direction for classifier system research, these properties of XCS make it suitable for a wide range of reinforcement learning situations where generalization over states is desirable. Key words Classifier systems, strength, fitness, accuracy, mapping, generalizati...

This paper presents the basic results and ideas of dynamic programming as they relate most directly to the concerns of planning in AI. These form the theoretical basis for the incremental planning methods used in the integrated architecture Dyna. These incremental planning methods are based on continually updating an evaluation function and the situation-action mapping of a reactive system. Actions are generated by the reactive system and thus involve minimal delay, while the incremental planning process guarantees that the actions and evaluation function will eventually be optimal -- no matter how extensive a search is required. These methods are well suited to stochastic tasks and to tasks in which a complete and accurate model is not available. For tasks too large to implement the situation-action mapping as a table, supervised-learning methods must be used, and their capabilities remain a significant limitation of the approach.

This paper reports and extends discussions carried out during a workshop held at the Santa Fe Institute in August 1995 by the authors. It treats eight major topics: (i) the importance of carefully examining research on routine, (it) the concept of 'action patterns ' in general and in terms of routine, (Hi) the useful categorization of routines and other recurring patterns, (iv) the research implications of recent cognitive results, (v) the relation of evolution to action patterns, (vi) the contributions of simulation modeling for theory in this area, (vii) examples of various approaches to empirical jj; research that reveal key problems, and (viii) a possible definition of 'routine'. An m extended appendix by Massimo Egidi provides a lexicon of synonyms and opposites ji covering use of the word 'routine ' in such areas as economics, organization theory and z artificial intelligence. 6

This paper builds on a previous review of significant research on adaptive behavior in animats. It summarizes the current state of the art and suggests some directions likely to provide interesting results in the near future. 1 Introduction An animat is a simulated animal or a real robot whose rules of behavior are inspired by those of animals. It is usually equipped with sensors, with actuators, and with a behavioral control architecture that allow it to react or to respond to variations in its environment (internal or external), notably to those that might impair its chances of survival. The behavior of an animat is what the animat does. This is characterized by a sequence of actions which reflects the dynamic interplay between the animat and its environment, mediated through the animat's sensors and actuators. The behavior of an animat is adaptive so long as it allows the animat to survive or to fulfill its mission. This requires that the animat's essential variables be monitored a...

Genetic algorithms are computational models of evolution that play a central role in many artificial-life models. We review the history and current scope of research on genetic algorithms in artificial life, using illustrative examples in which the genetic algorithm is used to study how learning and evolution interact, and to model ecosystems, immune system, cognitive systems, and social systems. We also outline a number of open questions and future directions for genetic algorithms in artificial-life research.

Diverse theories of animal navigation aim at explaining how to determine and maintain a course from one place to another in the environment, although each presents a particular perspective with its own terminologies. These vocabularies sometimes overlap, but unfortunately with different meanings. This paper attempts to precisely define the existing concepts and terminologies, so as to comprehensively describe the different theories and models within the same unifying framework. We present navigation strategies within a 4 level hierarchical framework based upon levels of complexity of required processing (Guidance, Place recognition-triggered Response, Topological navigation, Metric navigation). This classification is based upon what information is perceived, represented and processed. It contrasts with common distinctions based upon availability of certain sensors or cues and rather stresses the information structure and content of central processors. We then review computat...

Abstract not found

This paper adds a new viewpoint to the Anticipatory Classifier System (ACS). It approaches the system from a psychological perspective and thus provides new insights to the current system. The main learning mechanism in the ACS, the Anticipatory Learning Process (ALP), evolved out of the psychological learning theory of anticipatory behavioral control. The paper compares the ALP directly to this theory and reveals the similarities. Moreover, it investigates the behavior of the ACS. By simulating previously published rat experiments, the paper compares the behavior of the ACS with the behavior of the rats. Finally, two further cognitive mechanisms are introduced to the ACS. These two mechanisms result in an animal-like behavior of the ACS in the simulations. Furthermore, they prove the usability of the internal environmental model for reward-learning tasks for the first time.

XCS is a new kind of learning classifier system that differs from the traditional one primarily in its definition of classifier fitness and its relation to contemporary reinforcement learning. Advantages of XCS include improved performance and an ability to form accurate maximal generalizations. This paper reviews recent research on XCS with respect to representation, predictive modeling, internal state, noise, and underlying theory and technique. A notation for environmental regularities is introduced. 2 1 Introduction A classifier system is a learning system that seeks to gain reinforcement from its environment based on an evolving set of condition-action rules called classifiers. Via a Darwinian process, classifiers useful in gaining reinforcement are selected and propagate over those less useful, leading to increasing system performance. The classifier system idea is due to Holland (1986), who laid out a framework that included generalizability of classifier conditions, internal ...