Abstract

Learning in a multiagent system is a challenging problem due to two key factors. First, if other agents are simultaneously learning then the environment is no longer stationary, thus undermining convergence guarantees. Second, learning is often susceptible to deception, where the other agents may be able to exploit a learner’s particular dynamics. In the worst case, this could result in poorer performance than if the agent was not learning at all. These challenges are identifiable in the two most common evaluation criteria for multiagent learning algorithms: convergence and regret. Algorithms focusing on convergence or regret in isolation are numerous. In this paper, we seek to address both criteria in a single algorithm by introducing GIGA-WoLF, a learning algorithm for normalform games. We prove the algorithm guarantees at most zero average regret, while demonstrating the algorithm converges in many situations of self-play. We prove convergence in a limited setting and give empirical results in a wider variety of situations. These results also suggest a third new learning criterion combining convergence and regret, which we call negative non-convergence regret (NNR). 1

Citations

417	Markov games as a framework for multi-agent reinforcement learning - Littman - 1994
249	The dynamics of reinforcement learning in cooperative multiagent systems - Claus, Boutilier - 1998
237	Multiagent reinforcement learning: theoretical framework and an algorithm - Hu, Wellman - 1998
150	Multiagent learning using a variable learning rate - Bowling, Veloso
144	Gambling in a rigged casino: the adversarial multiarmed bandit problem - Auer, Cesa-Bianchi, et al. - 1995
140	A simple adaptive procedure leading to correlated equilibrium - Hart, Mas-Colell - 2000
125	Online convex programming and generalized infinitesimal gradient ascent - Zinkevich
100	Game Theory Evolving - Gintis - 2000
77	Nash convergence of gradient dynamics in general-sum games - Singh, Kearns, et al. - 2000
49	Correlated Q-learning - Greenwald, Hall - 2003
32	The extragradient method for finding saddle points and other problems - Korpelevich - 1977
28	Playing is believing: the role of beliefs in multi-agent learning - Chang, Kaelbling - 2002
5	Gunes Ercal, On no-regret learning, fictitious play, and nash equilibrium - Jafari, Greenwald, et al.