Learning to act in a multiagent environment is a difficult problem since the normal definition of an optimal policy no longer applies. The optimal policy at any moment depends on the policies of the other agents and so creates a situation of learning a moving target. Previous learning algorithms have one of two shortcomings depending on their approach. They either converge to a policy that may not be optimal against the specific opponents' policies, or they may not converge at all. In this article we examine this learning problem in the framework of stochastic games. We look at a number of previous learning algorithms showing how they fail at one of the above criteria. We then contribute a new reinforcement learning technique using a variable learning rate to overcome these shortcomings. Specifically, we introduce the WoLF principle, "Win or Learn Fast", for varying the learning rate. We examine this technique theoretically, proving convergence in self-play on a restricted class of iterated matrix games. We also present empirical results on a variety of more general stochastic games, in situations of self-play and otherwise, demonstrating the wide applicability of this method.

this article, please send e-mail to: tpami@computer.org, and reference IEEECS Log Number 108453

This paper investigates the problem of policy learning in multiagent environments using the stochastic game framework, which we briefly overview. We introduce two properties as desirable for a learning agent when in the presence of other learning agents, namely rationality and convergence. We examine existing reinforcement learning algorithms according to these two properties and notice that they fail to simultaneously meet both criteria. We then contribute a new learning algorithm, WoLF policy hillclimbing, that is based on a simple principle: "learn quickly while losing, slowly while winning." The algorithm is proven to be rational and we present empirical results for a number of stochastic games showing the algorithm converges.

We initiate the systematic study of algorithmic issues involved in finding equilibria (Nash and correlated) in games with a large number of players; such games, in order to be computationally meaningful, must be presented in some succinct, game-specific way. We develop a general framework for obtaining polynomial-time algorithms for optimizing over correlated equilibria in such settings, and show how it can be applied successfully to symmetric games (for which we actually find an exact polytopal characterization), graphical games, and congestion games, among others. We also present complexity results implying that such algorithms are not possible in certain other such games. Finally, we present a polynomial-time algorithm, based on quantifier elimination, for finding a Nash equilibrium in symmetric games when the number of strategies is relatively small.

Abstract not found

The fundamental distinction between ordinary evolutionary algorithms (EA) and co-evolutionary algorithms lies in the interaction between coevolving entities. We believe that this property is essentially game-theoretic in nature. Using game theory, we describe extensions that allow familiar mixing-matrix and Markov-chain models of EAs to address coevolutionary algorithm dynamics. We then employ concepts from evolutionary game theory to examine design aspects of conventional coevolutionary algorithms that are poorly understood.

This paper presents an overview of multi-agent system models of land-use/cover change (MAS/LUCC models). This special class of LUCC models combines a cellular landscape model with agent-based representations of decisionmaking, integrating the two components through specification of interdependencies and feedbacks between agents and their environment. The authors review alternative LUCC modeling techniques and discuss the ways in which MAS/LUCC models may overcome some important limitations of existing techniques. We briefly review ongoing MAS/LUCC modeling efforts in four research areas. We discuss the potential strengths of MAS/LUCC models and suggest that these strengths guide researchers in assessing the appropriate choice of model for their particular research question. We find that MAS/LUCC models are particularly well suited for representing complex spatial interactions under heterogeneous conditions and for modeling decentralized, autonomous decision making. We discuss a range of possible roles for MAS/LUCC models, from abstract models designed to derive stylized hypotheses to empirically detailed simulation models appropriate for scenario and policy analysis. We also discuss the challenge of validation and verification for MAS/LUCC models. Finally, we outline important challenges and open research questions in this new field. We conclude that, while significant challenges exist, these models offer a promising new tool for researchers whose goal is to create fine-scale models of LUCC phenomena that focus on human-environment interactions.

This work applies cluster analysis as a unified approach for a wide range of vision applications, thereby combining the research domain of computer vision and that of machine learning. Cluster analysis is the formal study of algorithms and methods for recovering the inherent structure within a given dataset. Many problems of computer vision have precisely this goal, namely to find which visual entities belong to an inherent structure, e.g. in an image or in a database of images. For example, a meaningful structure in the context of image segmentation is a set of pixels which correspond to the same object in a scene. Clustering algorithms can be used to partition the pixels of an image into meaningful parts, which may correspond to different objects. In this work we focus on the problems of image segmentation and image database organization. The visual entities to consider are pixels and images, respectively. Our first contribution in this work is a novel partitional (flat) clustering algorithm. The algorithm uses pairwise representation, where the visual objects (pixels,

We present a new energy-minimization framework for the graph isomorphism problem that is based on an equivalent maximum clique formulation. The approach is centered around a fundamental result proved by Motzkin and Straus in the mid-1960s, and recently expanded in various ways, which allows us to formulate the maximum clique problem in terms of a standard quadratic program. The attractive feature of this formulation is that a clear one-to-one correspondence exists between the solutions of the quadratic program and those in the original, combinatorial problem. To solve the program we use the so-called replicator equations—a class of straightforward continuous- and discrete-time dynamical systems developed in various branches of theoretical biology. We show how, despite their inherent inability to escape from local solutions, they nevertheless provide experimental results that are competitive with those obtained using more elaborate mean-field annealing heuristics.

Experimental results on the ultimatum game show clearly that (1) large fractions of players offer a 'fair' allocation and (2) that unfair (but positive) offers are systematically rejected. We offer an explanation of this behavior using the 'indirect evolutionary approach' which is based on the assumption that players behave rationally for given preferences but that their preferences change through an evolutionary process. We prove that despite anonymous interaction a preference for punishing unfair offers is an evolutionarily successful strategy if players interact in groups. This leads players to split the resource equally almost always. However, the equal split is not due to 'true fairness' (or 'altruism') but is entirely caused by the (justified) fear that unfair offers might be rejected. Our result can be interpreted as presenting an evolutionary foundation for the emergence of social norms. JEL-- classification numbers: C73, D 83. We thank Georg Noldeke, Karl Schlag,...