Abstract

This dissertation investigates the use of hierarchy and problem decomposition as a means of solving large, stochastic, sequential decision problems. These problems are framed as Markov decision problems (MDPs). The new technical content of this dissertation begins with a discussion of the concept of temporal abstraction. Temporal abstraction is shown to be equivalent to the transformation of a policy defined over a region of an MDP to an action in a semi-Markov decision problem (SMDP). Several algorithms are presented for performing this transformation efficiently. This dissertation introduces the HAM method for generating hierarchical, temporally abstract actions. This method permits the partial specification of abstract actions in a way that corresponds to an abstract plan or strategy. Abstr...

Citations

2960	Artificial Intelligence: A Modern Approach. 2nd edn - Russell, Norvig - 2003
2441	A robust layered control system for a mobile robot - Brooks - 1986
1965	Dynamic Programming - Bellman - 1957
1134	Reinforcement learning: A survey - Kaelbling, Littman, et al. - 1996
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
498	Markov decision processes - Puterman - 1995
478	Tsitsiklis. Parallel and Distributed Computation: Numerical Methods - Bertsekas, N - 1989
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
357	Generating project networks - Tate - 1977
307	Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition - Dietterich - 1999
250	Improving elevator performance using reinforcement learning - Crites, Barto - 1996
243	Acting optimally in partially observable stochastic domains - Cassandra, Kaelbling, et al. - 1994
223	Generalization in reinforcement learning: safely approximating the value function - Boyan, Moore - 1995
222	Motivated reinforcement learning - Dayan - 2002
207	Residual Algorithms: Reinforcement Learning with Function Approximation - Baird - 1995
202	Learning policies for partially observable environments: Scaling up - Littman, Cassandra, et al.
200	Exploiting structure in policy construction - Boutilier, Dearden, et al. - 1995
186	On the convergence of stochastic iterative dynamic programming algorithms - Jaakkola, Jordan, et al. - 1994
183	Teleo-reactive programs for agent control - Nilsson - 1994
178	Reinforcement Learning for Robots Using Neural Networks - Lin - 1993
172	Stable function approximation in dynamic programming - Gordon - 1995
158	Algorithms for sequential decision making - Littman - 1996
151	A survey of algorithmic methods for partially observed Markov decision processes - Lovejoy - 1991
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
150	Planning under time constraints in stochastic domains - DEAN, KAELBLING, et al.
150	Generalized nested dissection - LIPTON, ROSE, et al. - 1979
137	Reinforcement learning algorithm for partially observable Markov decision problems - Jaakkola, Singh, et al. - 1994
119	Discounted dynamic programming - Blackwell - 1965
114	On the complexity of solving Markov decision problems - Littman, Dean, et al. - 1995
112	Hierarchical solution of markov decision processes using macro-actions - Hauskrecht, Meuleau, et al. - 1998
105	Realtime learning and control using asynchronous dynamic programmming - Baro, Bradtke, et al. - 1991
103	Decomposition techniques for planning in stochastic domains - Dean, Lin - 1995
99	Reinforcement learning methods for continuous-time Markov decision problems - Bradtke, Duff - 1995
98	Finding structure in reinforcement learning - Thrun, Schwartz - 1995
97	Model minimization in Markov decision processes - Dean, Givan - 1997
80	Average reward reinforcement learning: Foundations, algorithms, and empirical results - Mahadevan - 1996
72	Iulti-time models for temporally abstract planning - Precup, Sutton - 1998
72	Tight performance bounds on greedy policies based on imperfect value functions - Williams, Baird - 1994
65	Using abstractions for decision-theoretic planning with time constraints - Boutilier, Dearden - 1994
65	Simulation-based optimization of Markov reward processes - Marbach, Tsitsiklis - 1998
51	Between mdps and semi-mdps: Learning, planning and representing knowledge at multiple temporal scales - Sutton, Precup, et al. - 1999
50	Decentralized supervisory control of discrete event systems - Lin, Wonham - 1988
48	Model reduction techniques for computing approximately optimal solutions for Markov decision processes - Dean, Givan, et al. - 1997
41	Self-improving factory simulation using continuous-time average-reward reinforcement learning - Mahadevan, Marchalleck, et al. - 1997
33	Markov Decision Processes - White - 1993
28	An improved policy iteration algorithm for partially observable MDPs - Hansen - 1997
28	Models of Delayed Reinforcement Learning - Watkins - 1989
27	Generalized prioritized sweeping - Andre, Friedman, et al. - 1997
22	Multilayer control of large Markov chains - Forestier, Varaiya - 1978
19	Neurogammon wins computer olympiad - Tesauro - 1990