We present an expressive agent design language for reinforcement learning that allows the user to constrain the policies considered by the learning process.The language includes standard features such as parameterized subroutines, temporary interrupts, aborts, and memory variables, but also allows for unspecified choices in the agent program. For learning that which isn’t specified, we present provably convergent learning algorithms. We demonstrate by example that agent programs written in the language are concise as well as modular. This facilitates state abstraction and the transferability of learned skills. 1

this paper we are concerned with how a user can write sequencer programs to effectively control the robot. Our emphasis is on issues of language and semantics: what is a good language for robot programs, what kind of semantics is appropriate for the sequencer, and how does the language fit the semantics. The result of our inquiries is the sequencer language Colbert, a part of the Sapphira architecture.

: For many years, research in AI plan generation was governed by a number of strong, simplifying assumptions: that the planning agent is omniscient, that its actions are deterministic and instantaneous, that its goals are fixed and categorical, and that its environment is static. More recently, researchers have developed expanded planning algorithms that are not predicated on such assumptions. But changing the way in which plans are formed is only part of what is required when the classical assumptions are abandoned. The demands of dynamic, uncertain environments mean that in addition to being able to form plans---even probabilistic, uncertain plans---agents must be able to effectively manage their plans. In this paper, which is based on a talk given at the 1998 AAAI Fall Symposium on Distributed, Continual Planning, we first identify reasoning tasks that are involved in plan management, including commitment management, environment monitoring, alternative assessment, plan elaboration, ...

One of the defining features of intelligent behaviour is the ordering of individual expressed actions into coherent, apparently rational patterns. Psychology has long assumed that hierarchical and sequential structures internal to the intelligent agent underlie this expression. Recently these assumptions have been challenged by claims that behaviour controlled by such structures is necessarily rigid, brittle, and incapable of reacting quickly and opportunistically to changes in the environment (Hendriks-Jansen 1996, Goldfield 1995, Brooks 1991a). This dissertation is intended to support the hypothesis that sequential and hierarchical structures are necessary to intelligent behaviour, and to refute the above claims of their impracticality. Three forms of supporting evidence are provided: • a demonstration in the form of experimental results in two domains that structured intelligence can lead to robust and reactive behaviour, • a review of recent research results and paradigmatic trends within artificial intelligence, and • a similar examination of related research in natural intelligence.

In this paper we describe a machine learning approach for acquiring a model of a robot behaviour from raw sensor data. We are interested in automating the acquisition of behavioural models to provide a robot with an introspective capability. We assume that the behaviour of a robot in achieving a task can be modelled as a finite stochastic state transition system. Beginning with data recorded by a robot in the execution of a task, we use unsupervised learning techniques to estimate a hidden Markov model (HMM) that can be used both for predicting and explaining the behaviour of the robot in subsequent executions of the task. We demonstrate that it is feasible to automate the entire process of learning a high quality HMM from the data recorded by the robot during execution of its task. The learned HMM can be used both for monitoring and controlling the behaviour of the robot. The ultimate purpose of our work is to learn models for the full set of tasks associated with a given problem domain, and to integrate these models with a generative task planner. We want to show that these models can be used successfully in controlling the execution of a plan. However, this paper does not develop the planning and control aspects of our work, focussing instead on the learning methodology and the evaluation of a learned model. The essential property of the models we seek to construct is that the most probable trajectory through a model, given the observations made by the robot, accurately diagnoses, or explains, the behaviour that the robot actually performed when making these observations. In the work reported here we consider a navigation task. We explain

The Plan Management Agent (PMA) is an intelligent software system that is intended to aid a user in managing a potentially large and complex set of plans. Currently under development, PMA applies AI technology for modeling and reasoning about plans and processes to the development of automated support for work activities. We have developed and implemented algorithms for reasoning about richly expressive plans, which include explicit temporal constraints, temporal uncertainty, and observation actions and conditional branches. We have also developed and implemented an approach to computing the cost of a new plan in the context of existing commitments. The current version of PMA has a low level of autonomy: it makes suggestions to its user, but it does not directly act on her behalf. In this paper, we first describe the PMA system, and then briefly raise some design questions we will need to address as we increase the level of PMA's autonomy, and have it vary with the situation. Introduc...

Physical schemas are representations of simple physically grounded relationships and interactions such as "move," "push," and "contain." We believe they are the conceptual primitives an agent employs to understand its environment. Physical schemas can be used at varying levels of abstraction across a variety of domains. We have designed a domain-general agent simulation and control testbed based on physical schemas. If a domain can be described in physical terms as agents moving and applying force, it can be simulated in this testbed. Furthermore, we show that physical schemas can be viewed as the basis for abstract plans and a domain-general planner, GRASP. Our simulation and planning system is currently being evaluated in a continuous, dynamic, and adversarial domain based on the game of Capture the Flag. The paper concludes with an example of how GRASP was applied to the problem of Course of Action generation and evaluation.

In this paper, we make a comparative review of a variety of "intermediate-level" robot languages that have emerged in recent years. We also describe a robot programming language called FROB (for Functional ROBotics). FROB is an example of an embedded, domain-specific language, hosted by the Haskell programming language.

) Martha E. Pollack Department of Computer Science and Intelligent Systems Program University of Pittsburgh Pittsburgh, PA 15260 USA pollack@cs.pitt.edu A significant amount of prior research effort in the field of Artificial Intelligence has gone into the design and analysis of planning algorithms. For the most part, this work has been guided by several strong, simplifying assumptions, most notably, that the plans will be performed in static, deterministic environments. Although these assumptions have made rigorous formal analysis possible, they make sense only for a limited number of applications, in which planning is done more or less in isolation of other reasoning tasks, and also in isolation of plan execution. Once we turn our attention to agents that perform autonomously in dynamic, uncertain environments---including multi-agent environments---the assumptions made by traditional planners are violated, and it becomes necessary to rethink the traditional AI approaches to planning....

Abstract – As a robot learns behaviors and task execution, several systems must be in place to allow the robot to store what has been learned as well as to recall learned information. We believe having a long-term memory is essential for our robot to be able to learn tasks and behaviors over time. We also believe that it is important for the robot to have some means of representing the immediate information in the environment. This goes beyond just a world representation and incorporates using short-term memory to track the environment. We have developed a robot that possesses such short-term and long-term memory systems. This robot has been given perceptive abilities with which it can populate its short-term memory and has been taught motion generation as a portion of its long-term memory population. Finally, through the use of a working memory system, we plan to show that our robot can “focus ” on pertinent taskrelated information from each of its memory systems in order to learn how to successfully execute tasks. Key Terms – short-term memory, long-term memory, working memory, behavior learning I.