Abstract

This paper presents a new approach to hierarchical reinforcement learning based on decomposing the target Markov decision process (MDP) into a hierarchy of smaller MDPs and decomposing the value function of the target MDP into an additive combination of the value functions of the smaller MDPs. The decomposition, known as the MAXQ decomposition, has both a procedural semantics---as a subroutine hierarchy---and a declarative semantics---as a representation of the value function of a hierarchical policy. MAXQ unifies and extends previous work on hierarchical reinforcement learning by Singh, Kaelbling, and Dayan and Hinton. It is based on the assumption that the programmer can identify useful subgoals and define subtasks that achieve these subgoals. By defining such subgoals, the programmer constrains the set of policies that need to be considered during reinforcement learning. The MAXQ value function decomposition can represent the value function of any policy that is consisten...

Citations

1964	Dynamic Programming - Bellman - 1957
1135	Learning from delayed rewards - Watkins - 1989
666	Soar: An architecture for general intelligence - Laird, Newell, et al. - 1987
649	Reinforcement learning - Sutton, Barto - 1998
554	Nonlinear Programming. Athena Scientific - Bertsekas - 1995
498	Markov Decision Processes - Puterman - 1994
494	Rete: A fast algorithm for the many pattern/many object pattern match problem - Forgy - 1982
472	Learning to act using real-time dynamic programming - Barto, Bradtke, et al. - 1995
434	Dynamic programming and Markov processes - Howard - 1960
411	Planning in a hierarchy of abstraction spaces - Sacerdoti - 1974
385	Learning and executing generalized robot plans - Fikes, Hart, et al. - 1972
342	Dynamic Programming and Optimal Control, Athena Scientific - Bertsekas - 2000
294	O-plan: the open planning architecture - Currie, Tate - 1991
275	Prioritized sweeping: Reinforcement learning with less data and less time - Moore, Atkeson - 1993
224	On-line Q-learning using connectionist systems - Rummery, Niranjan - 1994
222	Motivated reinforcement learning - Dayan - 2002
212	Reinforcement learning with hierarchies of machines - Parr, Russell - 1998
207	Residual Algorithms: Reinforcement Learning with Function Approximation - Baird - 1995
200	Exploiting structure in policy construction - Boutilier, Dearden, et al. - 1995
186	On the convergence of stochastic iterative dynamic programming algorithms - Jaakkola, Jordan, et al. - 1994
178	Reinforcement Learning for Robots Using Neural Networks - Lin - 1993
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
133	Decision theoretic planning: Structural assumptions and computational leverage - Boutilier, Dean, et al. - 1999
131	Macro operators: A weak method for learning - Korf - 1985
112	Hierarchical solution of markov decision processes using macro-actions - Hauskrecht, Meuleau, et al. - 1998
103	Decomposition techniques for planning in stochastic domains - Dean, Lin - 1995
102	The MAXQ method for hierarchical reinforcement learning - Dietterich - 1998
98	Hierarchical Control and Learning for Markov Decision Processes - Parr - 1998
91	Convergence results for single-step on-policy reinforcement-learning algorithms - Singh - 2000
78	Learning abstraction hierarchies for problem solving - Knoblock - 1990
72	Iulti-time models for temporally abstract planning - Precup, Sutton - 1998
57	Probabilistic Inference in Intelligent Systems: Networks of Plausible Inference - Pearl - 1988
55	TD models: Modeling the world at a mixture of time scales - Sutton - 1995
51	Between mdps and semi-mdps: Learning, planning and representing knowledge at multiple temporal scales - Sutton, Precup, et al. - 1999
42	Flexible decomposition algorithms for weakly coupled markov decision problems - Parr - 1998
35	Approximating value trees in structured dynamic programming - Boutilier, Dearden - 1996
25	Hierarchical reinforcement learning: Preliminary results - Kaelbling - 1993
20	Improved switching among temporally abstract actions - Sutton, Singh, et al. - 1998
19	Investigating production system representations for noncombinatorial match - Tambe, Rosenbloom - 1994
15	Multivalue-functions: Efficient automatic action hierarchies for multiple goal MDPs - Moore, Baird, et al. - 1999
15	Hierarchical explanation-based reinforcement learning - Tadepalli, Dietterich - 1997
14	l\lodule based reinforcement learning for a real robot - Kalmar, Lorincz - 1997
5	Economic principles of multi-agent systems (Editorial - Boutilier, Shoham, et al. - 1997
1	Multi-value-functions: E cient automatic action hierarchies for multiple goal MDPs - Moore, Baird, et al. - 1999
1	Hierarchical reinforcement learning: Preliminary results - Kaufmann, Francisco, et al. - 1993