From Reactive Behaviour to Adaptive Behaviour Motivational models for behaviour in animals and robots E. H. Spier, Balliol College, Trinity Term 1997 A thesis submitted for the degree of Doctor of Philosophy This thesis presents one possible way to design a control architecture that can be used to govern artificial animals. Such artefacts perform multiple-tasks and are expected to exist in a somewhat hostile environment -- they have to be adaptive. It also defends the position that automata, and animals, need not use reasoning to perform intelligent behaviour. Drawing from an ethological conception of motivation, a mathematical framework was described, computer simulations performed and preliminary work on a real robot discussed. It was shown that a reactive motivational algorithm performs better than alternatives that use simplistic models of the world, in a multiple resource foraging task. The reactive motivational framework was then extended to encompass instrumental behaviour as ...

To make progess in understanding human visuo-motor behavior, we will need to understand its basic components at an abstract level. One way to achieve such an understanding would be to create a model of a human that has a sufficient amount of complexity so as to be capable of generating such behaviors. Recent technological advances have been made that allow progress to be made in this direction. Graphics models that simulate extensive human capabilities can be used as platforms from which to develop synthetic models of visuo-motor behavior. Currently such models can capture only a small portion of a full behavioral repertoire, but for the behaviors that they do model, they can describe complete visuo-motor subsystems at a useful level of detail. The value in doing so is that the body’s elaborate visuo-motor structures greatly simplify the specification of the abstract behaviors that guide them. The net result is that, essentially, one is behaviors at each instant. This paper outlines one such model. A centerpiece of the model uses vision to aid the behavior that has the most to gain from taking environmental measurements. Preliminary tests of the model against human performance in realistic VR environments show that main features of the model show up in human behavior. Categories and Subject Descriptors: I.2.10 [Vision and Scene Understanding]: Perceptual reasoning 1.

Wilson's XCS is a clear departure from earlier classifier systems in terms of the way it calculates the fitness of classifiers for use in the genetic algorithm. Despite the growing body of work on XCS and the advantages claimed for it there has been no detailed comparison of XCS and traditional strength-based systems. This work takes a step towards rectifying this situation by surveying a number of issues related to the change in fitness. I distinguish different definitions of overgenerality for strength and accuracy-based fitness and analyse some implications of the use of accuracy, including an apparent advantage in addressing the explore/exploit problem. I analyse the formation of strong overgenerals, a major problem for strength-based systems, and illustrate their dependence on biased reward functions. I consider motivations for biasing reward functions in single step environments, and show that non-trivial multi step environments have biased Q-functions. I conclude that XC...

: `Pigs and people' is a simulated environment, in which action selection mechanisms can be evaluated and compared. Action selection mechanisms attempt to solve the action selection problem faced by both animals and robots: the problem of selecting which actions to perform in order to achieve heterogeneous and possibly conflicting goals. In this project, two action selection mechanisms are implemented: the non-learning drives mechanism, and W-learning. The non-learning mechanism considerably outperforms the learning mechanism. The results achieved by the two mechanisms are compared and analysed in an attempt to explain the difference in performance.

Agents are of interest mainly when confronted with complex tasks. We propose a methodology for the automated design of such agents (in the framework of Markov Decision Processes) in the case where the global task can be decomposed into simpler-possibly concurrent- sub-tasks. This is accomplished by automatically combining basic behaviors using Reinforcement Learning methods. The main idea is to build a global policy using a weighted combination of basic policies, the weights being learned by the agent (using Simulated Annealing in our case). These basic behaviors can either be learned or reused from previous tasks since they will not need to be tuned to the new task. Furthermore, the agents designed by our methodology are highly scalable as, without further refinement of the global behavior, they can automatically combine several instances of the same basic behavior to take into account concurrent occurences of the same subtask.

On behalf of:

It is proposed that researchers in AI and ALife construct their agent minds and agent worlds as servers on the Internet. Under this scheme, not only will 3rd parties be able to re-use agent worlds in their own projects (a long-standing aim of other schemes), but 3rd parties will be able to re-use agent minds as components in larger, multiple-mind, cognitive systems. Under this scheme, any 3rd party user on the Internet may select multiple minds from different remote "mind servers", select a remote "Action Selection server" to resolve the conflicts between them, and run the resulting "society of mind" in the world provided on another "world server". Re-use is done not by installing the software, but rather by using a remote service. Hence the term, the "World-Wide-Mind" (WWM), referring to the fact that the mind may be physically distributed across the world. This model addresses the possibility that the AI project may be too big for any single laboratory to complete, so it will be necessary both to decentralise the work and to allow a massive and ongoing experiment with different schemes of decentralisation. We expect that researchers will not agree on how to divide up the AI work, so components will overlap and be duplicated and we need multiple-conflicting-minds models [21].

This paper introduces an integration of reinforcement learning and behavior-based control designed to produce real-time learning in situated agents. The model layers a distributed and asynchronous reinforcement learning algorithm over a learned topological map and standard behavioral substrate to create a reinforcement learning complex. The topological map creates a small and task-relevant state space that aims to make learning feasible, while the distributed and asynchronous aspects of the architecture make it compatible with behavior-based design principles.

In the first part of this paper, a change in methodology for the future of AI and Adaptive Behavior research is proposed. It is proposed that researchers construct their agent minds and their agent worlds as servers on the Internet. 3rd parties will use these servers as components in larger systems. In this scheme, any user on the Internet will be able to (a) select multiple minds from different remote "mind servers", (b) select a remote "Action Selection server" to resolve the (inevitable) conflicts between these minds, and (c) run the resulting constructed "society of mind" in the world provided on another "world server". All this without necessarily having to consult with the server authors. This constructed society may now also be presented as just another primitive mind server, ready for reuse by others as a component in a larger system. From the current situation of isolated experiments we will move to a situation where not only can researchers use each other's agent worlds, but they can also use each other's agent minds as components in larger systems. Servers may call other servers, and it is expected that 3rd parties will continuously write wrappers and filters for existing mind servers, overriding and modifying their default behaviour (to produce new, co-existing mind servers). None of this necessarily means that the mind being used ever leaves its server (or that its insides are even made public). Hence the term, the "World-Wide-Mind" (WWM), referring to the fact that the mind may be physically distributed across the world, with parts of the mind at different remote servers. Part of the motivation for the WWM is that if the AI project is to be successful, it may be too big for any single laboratory to complete. So it will be necessary both to decentralise t...

The `Radical Agent Concept' in this chapter is that communication between agents in a MAS should be the simplest part of the system. When extensive real-time coordination between modules is required, then those modules should probably be considered elements of a single modular agent rather than as agents themselves. The advantage of this distinction is that system developers can then leverage standard software-engineering practices and more centralized coordination mechanisms to reduce the over-all complexity of the system. In this chapter I provide arguments for this point and also examples, both from nature and from my own research in building modular agents.