DMCA
Internal-state policy-gradient algorithms for partially observable Markov decision processes (2002)
Cached
Download Links
| Citations: | 5 - 1 self |
BibTeX
@TECHREPORT{Aberdeen02internal-statepolicy-gradient,
author = {Douglas Aberdeen and Jonanthan Baxter},
title = {Internal-state policy-gradient algorithms for partially observable Markov decision processes},
institution = {},
year = {2002}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Policy-gradient algorithms are attractive as a scalable approach to learning approximate policies for controlling partially observable Markov decision processes (POMDPs). POMDPs can be used to model a wide variety of learning problems, from robot navigation to speech recognition to stock trading. The downside of this generality is that exact algorithms are computationally intractable, motivating approximate methods. Existing policy-gradient methods have worked well for problems admitting good memory-less solutions, but have failed to scale to large problems requiring memory. This paper develops novel algorithms for learning policies with memory. We present a new policy-gradient algorithm that uses an explicit model of the POMDP to estimate gradients, and demonstrate its effectiveness on problems with tens of thousands of states. We also describe three new Monte-Carlo algorithms that learn by interacting with their environment. We compare these algorithms on non-trivial POMDPs, including noisy robot navigation and multi-agent settings.
Keyphrases
observable markov decision process internal-state policy-gradient algorithm large problem non-trivial pomdps approximate method explicit model noisy robot navigation multi-agent setting approximate policy new policy-gradient algorithm stock trading policy-gradient algorithm robot navigation exact algorithm wide variety policy-gradient method new monte-carlo algorithm good memory-less solution scalable approach novel algorithm
