DMCA
An analysis of temporal-difference learning with function approximation (1997)
Cached
Download Links
| Venue: | IEEE Transactions on Automatic Control |
| Citations: | 313 - 8 self |
BibTeX
@TECHREPORT{Tsitsiklis97ananalysis,
author = {John N. Tsitsiklis and Benjamin Van Roy},
title = {An analysis of temporal-difference learning with function approximation},
institution = {IEEE Transactions on Automatic Control},
year = {1997}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We discuss the temporal-difference learning algorithm, as applied to approximating the cost-to-go function of an infinite-horizon discounted Markov chain. The algorithm weanalyze updates parameters of a linear function approximator on-line, duringasingle endless trajectory of an irreducible aperiodic Markov chain with a finite or infinite state space. We present a proof of convergence (with probability 1), a characterization of the limit of convergence, and a bound on the resulting approximation error. Furthermore, our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal-difference learning. In addition to proving new and stronger positive results than those previously available, we identify the significance of on-line updating and potential hazards associated with the use of nonlinear function approximators. First, we prove that divergence may occur when updates are not based on trajectories of the Markov chain. This fact reconciles positive and negative results that have been discussed in the literature, regarding the soundness of temporal-difference learning. Second, we present anexample illustrating the possibility of divergence when temporal-difference learning is used in the presence of a nonlinear function approximator.
Keyphrases
temporal-difference learning function approximation markov chain algorithm weanalyze new intuition temporal-difference learning algorithm nonlinear function approximators approximation error linear function approximator cost-to-go function on-line updating irreducible aperiodic markov chain infinite state space nonlinear function approximator potential hazard positive result new line negative result duringasingle endless trajectory
