Abstract

We consider the problem of solving a nonhomogeneous infinite horizon Markov Decision Process (MDP) problem in the general case of potentially multiple optimal first period policies. More precisely, we seek an algorithm that, given a finite subset of the problem’s potentially infinite data set, delivers an optimal first period policy. Such an algorithm can thus recursively generate, within a rolling horizon procedure, an infinite horizon optimal solution to the original infinite horizon problem. However it can happen that for a given problem no such algorithm exists. In this case, it is impossible to solve the problem with a finite amount of data. We say such problems fail to be wellposed. Under the assumption of increasing marginal returns in actions (with respect to states) and stochastically increasing states into which the system transitions (with respect to actions), we provide an algorithm that is guaranteed to solve the corresponding nonhomogeneous MDP whenever the problem is well-posed. The algorithm proceeds by discovering, in finite time, a forecast horizon for which an optimal solution delivers an optimal first period policy to the infinite horizon problem. In particular, we show by construction, the existence of a forecast horizon (and hence, a solution horizon) for all such well-posed problems. We illustrate the theory and algorithms developed by solving every well-posed instance of the time-varying version of the classic asset selling problem.

Citations