Abstract

On large problems, reinforcement learning systems must use parameterized function approximators such as neural networks in order to generalize between similar situations and actions. In these cases there are no strong theoretical results on the accuracy of convergence, and computational results have been mixed. In particular, Boyan and Moore reported at last year's meeting a series of negative results in attempting to apply dynamic programming together with function approximation to simple control problems with continuous state spaces. In this paper, we present positive results for all the control tasks they attempted, and for one that is significantly larger. The most important differences are that we used sparse-coarse-coded function approximators (CMACs) whereas they used mostly global function approximators, and that we learned online whereas they learned offline. Boyan and Moore and others have suggested that the problems they encountered could be solved by using actual outcomes (...

Citations

1135	Learning from delayed rewards - Watkins - 1989
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
334	Practical issues in temporal difference learning - Tesauro - 1992
261	Robot Dynamics and Control - Spong, Vidyasagar - 1989
255	Self-improving reactive agents based on reinforcement learning, planning, and teaching - Lin - 1992
250	Improving elevator performance using reinforcement learning - Crites, Barto - 1996
224	On-line Q-learning using connectionist systems - Rummery, Niranjan - 1994
223	Generalization in reinforcement learning: safely approximating the value function - Boyan, Moore - 1995
219	Temporal Credit Assignment in Reinforcement Learning - Sutton - 1984
207	Residual Algorithms: Reinforcement Learning with Function Approximation - Baird - 1995
172	Stable function approximation in dynamic programming - Gordon - 1995
166	Reinforcement learning with replacing eligibility traces - Singh, Sutton - 1996
157	Neuronlike elements that can solve difficult learning control problems - Barto, Sutton, et al. - 1983
105	Realtime learning and control using asynchronous dynamic programmming - Baro, Bradtke, et al. - 1991
58	The convergence of TD() for general - Dayan - 1992
52	CMAC: An associative neural network alternative to backpropagation - Miller, Glanz, et al. - 1990
38	Online learning with random representations - Sutton, Whitehead - 1993
32	The convergence of TD(λ) for general λ - Dayan - 1992
23	CMAC-based adaptive critic self-learning control - Lin, Kim - 1991
21	Feature-based methods for large-scale dynamic programming - Tsitsiklis, Roy - 1994
20	A counterexample to temporal differences learning - Bertsekas - 1995
14	Reinforcement learning for planning and Control - Dean, Basye, et al. - 1993
10	Swinging up the acrobot: An example of intelligent control - DeJong, Spong - 1994
9	Modular On-line Function Approximation for Scaling up Reinforcement Learning - Tham - 1994