Citation Context

... the use of eligibility traces but we believe the extension to that case would be straightforward. 2 The Policy Gradient Framework We consider the standard reinforcement learning framework (e.g., see =-=Sutton & Barto, 1998-=-), in which a learning agent interacts with a stochastic environment and this interaction is modeled as a discrete-time Markov decision process. The state, action, and reward at each time t ∈ {0, 1, 2...

Citation Context

...A variety of methods can be used to solve the prediction problem, but the ones that have proved most effective in large applications are those based on some form of temporal difference (TD) learning (=-=Sutton, 1988-=-) in which estimates are updated on the basis of other estimates. Such bootstrapping methods can be viewed as a way of accelerating learning by trading bias for variance. Actor-critic methods were amo...

Citation Context

...ng methods can be viewed as a way of accelerating learning by trading bias for variance. Actor-critic methods were among the earliest to be investigated in reinforcement learning (Barto et al., 1983; =-=Sutton, 1984-=-). They were largely supplanted in the 1990’s by methods that estimate action-value functions and use them directly to select actions without an explicit policy structure. This approach was appealing ...

Citation Context

...re to converge. These problems led to renewed interest in methods with an explicit representation of the policy, which came to be known as policy gradient methods (Marbach, 1998; Sutton et al., 2000; =-=Konda & Tsitsiklis, 2000-=-; Baxter & Bartlett, 2001). Policy gradient methods without bootstrapping can be easily proved convergent, but converge slowly because of the high variance of their gradient estimates. Combining them ...

Citation Context

...lems led to renewed interest in methods with an explicit representation of the policy, which came to be known as policy gradient methods (Marbach, 1998; Sutton et al., 2000; Konda & Tsitsiklis, 2000; =-=Baxter & Bartlett, 2001-=-). Policy gradient methods without bootstrapping can be easily proved convergent, but converge slowly because of the high variance of their gradient estimates. Combining them with bootstrapping is a p...

Citation Context

...es. Such bootstrapping methods can be viewed as a way of accelerating learning by trading bias for variance. Actor-critic methods were among the earliest to be investigated in reinforcement learning (=-=Barto et al., 1983-=-; Sutton, 1984). They were largely supplanted in the 1990’s by methods that estimate action-value functions and use them directly to select actions without an explicit policy structure. This approach ...

Citation Context

... of the average reward can be written as ∇J(π) = � d π (s) � ∇π(a|s)(Q π (s, a) ± b(s)). (3) s∈S a∈A The baseline can be chosen such in a way that the variance of the gradient estimates is minimized (=-=Greensmith et al., 2004-=-). The natural gradient, denoted ˜ ∇J(π), can be calculated by linearly transforming the regular gradient, using the inverse Fisher information matrix of the policy: ˜ ∇J(π) = G −1 (θ)∇J(π). The Fishe...

Citation Context

...ties including in some cases a failure to converge. These problems led to renewed interest in methods with an explicit representation of the policy, which came to be known as policy gradient methods (=-=Marbach, 1998-=-; Sutton et al., 2000; Konda & Tsitsiklis, 2000; Baxter & Bartlett, 2001). Policy gradient methods without bootstrapping can be easily proved convergent, but converge slowly because of the high varian...