Abstract

In this paper, we first present a key approximation result for zero-sum, discounted Markov games, providing bounds on the state-wise loss and the loss in the sup norm resulting from using approximate Q-functions (e.g., Q-functions estimated by sampling). Then, we extend the policy rollout sampling technique for MDPs to Markov games. Using our key approximation result, we prove that under certain conditions, the resulting rollout technique for games gives rise to a policy that is closer to the Nash equilibrium than the base policy, using an amount of sampling completely independent of the state space size. We also use our key result to provide an alternative to a published analysis of a second sampling approach to Markov games known as "sparse sampling". Thus, our theorem implies the (already known) result that, under certain conditions, the policy generated by the sparse-sampling algorithm is close to the Nash equilibrium. Again, the amount of sampling that guarantees the result is independent of the size of the state space of the Markov game. We also provide simulation results to demonstrate the practicality of our extension of the rollout technique.

Citations