Abstract

On large problems, reinforcement learning systems must use parameterized function approximators such as neural networks in order to generalize between similar situations and actions. In these cases there are no strong theoretical results on the accuracy of convergence, and computational results have been mixed. In particular, Boyan and Moore reported at last year's meeting a series of negative results in attempting to apply dynamic programming together with function approximation to simple control problems with continuous state spaces. In this paper, we present positive results for all the control tasks they attempted, and for one that is significantly larger. The most important differences are that we used sparse-coarse-coded function approximators (CMACs) whereas they used mostly global function approximators, and that we learned online whereas they learned offline. Boyan and Moore and others have suggested that the problems they encountered could be solved by using actual outcomes (...

Citations

1235	Learning from delayed rewards - Watkins - 1989
1165	Learning to predict by the methods of temporal differences - Sutton - 1988
356	Practical issues in temporal difference learning - Tesauro - 1992
288	Robot Dynamics and Control - Spong, Vidyasagar - 1989
269	A.G.: Improving elevator performance using reinforcement learning - Crites, Barto - 1996
269	Self-improving reactive agents based on reinforcement learning, planning and teaching - Lin - 1992
259	On-line Q-learning using connectionist systems,” Engineering Department - Rummery, Niranjan - 1994
238	Generalization in reinforcement learning: Safely approximating the value function - Boyan, Moore - 1994
231	Temporal credit assignment in reinforcement learning - Sutton - 1984
222	Residual algorithms: Reinforcement learning with function approximation - Baird - 1995
195	Stable function approximation in dynamic programming - Gordon - 1995
178	Reinforcement learning with replacing eligibility traces - Singh, Sutton - 1996
172	Neuron-like elements that can solve difficult learning control problems - Barto, Sutton, et al. - 1983
108	Real-time Learning and control using Asynchronous Dynamic Programming - Barto, Bradtke, et al. - 1991
62	L.G.: Cmac: An associative neural network alternative to backpropagation - Miller, Glanz, et al. - 1990
61	The convergence of td() for general - Dayan - 1992
46	Online learning with random representations - Sutton, Whitehead - 1993
36	The convegence of TD(λ) for general λ - Dayan - 1992
25	CMAC-based adaptive critic self-learning control - Lin, Hyongsuk - 1991
25	Feature-based methods for large-scale dynamic programming - JN, Roy - 1996
22	A counter-example to temporal differences learning. Neural Computation 7:270–279 - Bertsekas - 1994
17	Reinforcement learning for planning and control - Dean, Basye, et al. - 1993
13	Swinging up the acrobot: An example of intelligent control - 16DeJong, Spong - 1994
10	Modular On-Line Function Approximation for Scaling Up Reinforcement Learning - Tham - 1994