Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 673--682, 2001.  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... these problems are used repeatedly in solving problem variants such as stochastic games, and partially observable MDPs [Sha53, Han98] Preferred methods for solving MDP problems use dynamic programming techniques, and in particular often contain a so called value iteration or policy iteration loop [Put94, Lit96, Han98, GKP01]. These methods converge to optimal solutions quickly in practice, but we know little about their asymptotic complexity. It is known, however, that algorithms based on value iteration have no better than a pseudo polynomial run time on MDP prob An algorithm has pseudo polynomial run time ....

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In AAAI, pages 673-- 679, 2001.

.... these problems are used repeatedly in solving problem variants such as stochastic games, and partially observable MDPs [Sha53, Han98] Preferred methods for solving MDP problems use dynamic programming strategies, and in particular often contain a so called value iteration or policy iteration loop [Put94, Lit96, Han98, GKP01]. These methods converge to optimal solutions quickly in practice, but we know little about their asymptotic complexity. It is known, however, that algorithms based on value iteration have no better than a pseudo polynomial run time on MDP prob An algorithm has pseudo polynomial run time ....

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In AAAI, pages 673-- 679, 2001.

....observable states, where the states are discrete and the state space is relatively small. However, many cases of interest involve continuous or partially observable states, which pose di#cult challenges for these methods. Whereas much recent work has attempted to meet these challenges (see, e.g. [4 7]) planning under uncertainty remains an open problem and the subject of active research. This paper presents and demonstrates a new approach to selecting an optimal action sequence. It is motivated by the observation, which is perhaps a bit controversial, that the traditional approach may be ....

C. Guestrin, D. Koller, R. Parr (2001). Max-norm projections for factored MDPs. Proc. IJCAI-01, vol. 1, 673-680.

....1 Introduction Dynamic programming approaches to finding optimal control policies in Markov decision processes (MDPs) 4, 14] using explicit (flat) state space representations break down when the state space becomes extremely large. More recent work extends these algorithms to use propositional [6, 11, 7, 12] as well as relational [8] state space representations. These extensions have not yet shown the capacity to solve large classical planning problems such as the benchmark problems used in planning competitions [2] These methods typically calculate a sequence of cost functions. For familiar ....

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In IJCAI, pages 673--680, 2001.

....of S. There are O(jSj) constraints and jSjj S j unknown entries in matrix F . We describe several techniques that allow one to exploit problem structure to find an acceptable lossy compression without state space enumeration. One approach is related to the basis function model proposed in [4], in which we restrict F to be functions over some small set of factors (subsets of state variables. This ensures that the number of unknown parameters in any column of F (which we optimize in Table 1) is Assuming S is small, the j Sj variables in each and j Sj variables in R ....

....manageable set of unknowns. To deal with the O(jSj) constraints, we can exploit the structure imposed on F and the DBN structure to reduce the number of constraints to something (in the many cases) polynomial in the number of state variables. This can be achieved using the techniques described in [4, 16] to rewrite an LP with many fewer constraints or to generate small subsets of constraints incrementally. These techniques are rather involved, so we refer to the cited papers for details. By searching within a restricted set of structured compressions and by exploiting DBN structure it is ....

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. Proc. IJCAI-01, pp.673--680, Seattle, WA, 2001.

....techniques that can find approximately optimal linear approximators in way that exploits the structure of the MDP without enumerating state space. We assume that each basis function f j is compact, referring only to a small set of variables X j . Linear value and policy iteration are described in [8] , while a factored LP solution technique is presented in [16; 9] We discuss the method proposed in [16] The LP formulation of a factored MDP above can be encoded compactly when an MDP is factored. First, notice that the objective function Eq. 3 can be encoded compactly: w j f j (x) ....

.... basis function f j can exploit the fact that it refers only a small subset of variables; the regression of f j through a produces a function that includes only those variables X j , and variables in X k [3] The maximization over x is nonlinear, but can encoded using the clever trick of [8] . For a fixed set of weights, a cost network can be solved using variable elimination to determine this max without state space enumeration. While this technique scales exponentially with the maximum number of variables in any function (i.e, the functions f j , B f j ) orinter mediate ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proc. Seventeenth International Joint Conf. on AI, pp.673--680, Seattle, 2001.

....Neither of these two assumptions alone is sucient to permit ecient policy optimization for large MDPs. However, combined, the two assumptions allow approximate solutions to be obtained for problems involving trillions of states reasonably quickly. 3. 1 Factored MDPs In the spirit of [7, 8, 6] we de ne a factored MDP to be one that can be represented compactly by an additive reward function and a factored state transition model. Speci cally, we assume the reward function decomposes as R(x; a) P m R a;r (x a;r ) where each local reward function R a;r is de ned on a small set of ....

....value function rather than calculate it exactly. Numerous schemes have been investigated for approximating optimal value functions and policies in a compact representational framework, including: hierarchical decompositions [5] decision trees and diagrams [3, 12] generalized linear functions [1, 13, 4, 7, 8, 6], neural networks [2] and products of experts [11] However, the simplest of these is generalized linear functions, which is the form we investigate below. In this case, we consider functions of the form f(x) P k w j b j (x j ) where b 1 ; b k are a xed set of basis functions, and x j ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projection for factored MDPs. In Proceedings IJCAI, 2001.

....Neither of these two assumptions alone is sucient to permit ecient policy optimization for large MDPs. However, combined, the two assumptions allow approximate solutions to be obtained for problems involving trillions of states reasonably quickly. 3. 1 Factored MDPs In the spirit of [7, 8, 6] we de ne a factored MDP to be one that can be represented compactly by an additive reward function and a factored state transition model. Speci cally, we assume the reward function decomposes as R(x; a) P m r=1 R a;r (x a;r ) where each local reward function R a;r is de ned on a small set of ....

....value function rather than calculate it exactly. Numerous schemes have been investigated for approximating optimal value functions and policies in a compact representational framework, including: hierarchical decompositions [5] decision trees and diagrams [3, 12] generalized linear functions [1, 13, 4, 7, 8, 6], neural networks [2] and products of experts [11] However, the simplest of these is generalized linear functions, which is the form we investigate below. In this case, we consider functions of the form f(x) P k j=1 w j b j (x j ) where b 1 ; b k are a xed set of basis functions, and ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projection for factored MDPs. In Proceedings IJCAI, 2001.

....Neither of these two assumptions alone is sucient to permit ecient policy optimization for large MDPs. However, combined, the two assumptions allow approximate solutions to be obtained for problems involving trillions of states reasonably quickly. 3. 1 Factored MDPs In the spirit of [7, 8, 6] we de ne a factored MDP to be one that can be represented compactly by an additive reward function and a factored state transition model. Speci cally, we assume the reward function decomposes as R(x; a) P m r=1 R a;r (x a;r ) where each basis function R a;r is de ned on a small set of ....

....value function rather than calculate it exactly. Numerous schemes have been investigated for approximating optimal value functions and policies in a compact representational framework, including: hierarchical decompositions [5] decision trees and diagrams [3, 12] generalized linear functions [1, 13, 4, 7, 8, 6], neural networks [2] and products of experts [11] However, the simplest of these is generalized linear functions, which is the form we investigate below. In this case, we consider functions of the form f(x) P k j=1 w j b j (x j ) where b 1 ; b k are a xed set of basis functions, and ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projection for factored MDPs. In Proceedings IJCAI (to appear), 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm Projections for Factored MDPs. International Joint Conference on Artificial Intelligence, 17:673--680, 2001.

No context found.

C. E. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI, 2001.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proc. of IJCAI-01, 2001.

No context found.

Guestrin, C.; Koller, D.; and Parr, R. 2001a. Max-norm projection for factored MDPs. In Proceedings IJCAI.

No context found.

C. E. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI, 2001.

No context found.

C. E. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proc. Seventeenth International Joint Conference on Artificial Intelligence (IJCAI-01), pages 673 -- 680, 2001.

No context found.

C. E. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI, 2001.

No context found.

C. E. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI, 2001.

No context found.

C. E. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI-01, 2001.

....have proposed the use of a linear approximation, where an approximate value function is represented as a linear combination of basis function. This approach was first proposed for a variety of unfactored MDPs [Tsitsiklis and Van Roy, 1996] and applied to factored MDPs in [Koller and Parr, 2000; Guestrin et al. 2001] They show that even a small set of basis functions can provide a high quality approximation to a high dimensional value function. In this paper, we apply this idea to POMDPs, by using the same approximation for the individual value function vectors that comprise the POMDP value function. In ....

....viewed as vectors. Our approximate value function is then represented by Aw. The idea of using linear value functions for dynamic programming was proposed, initially, by Bellman et al. 1963] and has been further explored recently [Tsitsiklis and Van Roy, 1996; Koller and Parr, 1999; 2000; Guestrin et al. 2001] . The basic idea is as follows: in the solution algorithms, whether value iteration or policy iteration, we use only value functions within H. Whenever the algorithm takes a step that results in a value function V that is outside this space, we project the result back into the space by finding ....

[Article contains additional citation context not shown here]

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI-01), Seattle, Washington, August 2001. Morgan Kaufmann.

....in this way, then we could use our algorithm of Section 3 to implement Greedy(V) by having the agents use our message passing coordination algorithm at each step. Here we have only one function h per agent, but our approach extends trivially to the case of multiple h functions. In previous work [9, 6], we presented algorithms for computing approximate value functions of this form for factored MDPs. These algorithms can circumvent the exponential blowup in the number of state variables, but explicitly enumerate the action space of the MDP, making them unsuitable for the exponentially large ....

....of the constraint can be viewed as the sum of restricted scope functions parameterized by w. For a fixed w, we can compute the maximum over fx; ag using a cost network, as in Section 2. If w is not specified, the maximization induces a family of cost networks parameterized by w. As we showed in [6], we can turn this cost network into a compact set of LP constraints on the free variable w. More generally, suppose we wish to enforce the constraint 0 max y F (y) where F (y) j (y) such that each f j has a restricted scope. Here, the superscript w indicates that each f j might be ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proc. 17th IJCAI, 2001.

....in the size of the state description. A polynomial size representation is often thought to be essential to scale up model based MDP planning to realistic problems [2] Of these approaches, the factored MDP representation developed by Koller and Parr has proven to be particularly convenient [17, 18, 12]. Combined with linear value function approximators it allows practically efficient planning algorithms based on linear programming to be easily implemented [13, 22] Both lines of research exploration exploitation and compact representations have recently been brought together in the ....

....provides some analysis of the error relative to that of the best possible approximation in the subspace. This transformation has the effect of reducing the number of free variables in the linear program to k 1, but the number of constraints remains X A . Fortunately, using the algorithms of [12, 13, 22] one can exploit the structure of a factored MDP (see Section 3) to obtain a compact representation and efficient solution to this linear program. The second limitation of the explicit MDP planning approach, unknown model parameters, has been the focus of extensive work in the field of ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proc. IJCAI, pages 673--682, 2001.

....the entire network. For a network of n machines, the number of states in this MDP is 9 (3 status levels 3 load levels per computer) and the joint action space contains 2 actions. We implemented our multiagent LSPI algorithm and tested it on a variety of network topologies, as defined in [7]. Fig. 2 shows the estimated value of the resulting policies for problems with increasing number of agents. For comparison, we also plot the results reported by Guestrin et al. 8] for three other methods: their LP based (LP) approach; and Schneider et al. s [13] Distributed Reward (DR) and ....

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In IJCAI-01, 2001.

....methods can be incorporated into any of a number of policy search methods to fine tune a policy derived by Q learning with linear Q functions or by LSPI. 7 Experimental results In this section we report results of applying our Coordinated RL approach with LSPI to the SysAdmin multiagent domain [7] . The SysAdmin problem consists of a network of n machines connected in one of the following topologies: chain, ring, star, ring of rings, or star andring. The state of each machine j is described by two variables: status S j # good, faulty, dead , and load L j # idle, loaded, process ....

....methods might be able to cope with the huge state space, but the problem of the huge action space remains. We apply our Coordinated RL approach with LSPI to jointly address the value function approximation and exponential action space problems. The SysAdmin problem has been studied in [7] , where the model of the process is assumed to be available as a factored MDP. The state value function is approximated as a linear combination of indicator basis functions, and the coefficients are computed using a Linear Programming (LP) approach. The derived policies are very close to the ....

[Article contains additional citation context not shown here]

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In AAAI Spring Symposium, Stanford, California, March 2001.

....X x 0 P (x 0 j x; a) X i w i h i (x 0 ) 8x 2 X;8a 2 A: Although there are exponentially many constraints, we can replace these constraints by an equivalent set which is exponentially smaller. In previous work, we have applied such transformation in the context of single agent problems [7] and table based multiagent factored MDPs [8] We will now extend these ideas to exploit the rule based representation of our reward and basis functions. First, note that the constraints above can be replaced by a single, nonlinear constraint: 0 max x;a R(x; a) X i ( g i (x) h i ....

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proc. of IJCAI-01, 2001.

....3 to implement Greedy(V) by having the agents use our message passing coordination algorithm at each step. For simplicity of presentation, we have only one function h per agent, but, as we mentioned, this extends trivially to the more general case of multiple h functions. In previous work [10, 7], we presented algorithms for computing approximate value functions of this form for factored MDPs. However, these algorithms explicitly iterated over the action space of the MDP, making them unsuitable for the exponentially large action space in multiagent MDPs. We now provide a novel algorithm ....

.... 0. For a factored MDP, the difference QV (s; a) V(s) can be viewed as a sum of restricted domain functions. If w is not specified, the maximization induces a family of cost networks parameterized by w. Our task is to turn these into a compact set of LP constraints on the free variable w. See [7] for a similar transformation. More generally, suppose we wish to enforce the constraint max y F w (y) 0, where F w (y) P j f w j (y) such that each f j has a restricted domain. Consider any function e used within the cost network to maximize F w , including the original f j s, ....

[Article contains additional citation context not shown here]

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proceedings of International Joint Conference on AI (IJCAI-01), 2001.

....have proposed the use of a linear approximation, where an approximate value function is represented as a linear combination of basis function. This approach was first proposed for a variety of unfactored MDPs [Tsitsiklis and Van Roy, 1996] and applied to factored MDPs in [Koller and Parr, 2000; Guestrin et al. 2001] They show that even a small set of basis functions can provide a high quality approximation to a high dimensional value function. In this paper, we apply this idea to POMDPs, by using the same approximation for the individual value function vectors that comprise the POMDP value function. In ....

....viewed as vectors. Our approximate value function is then represented by Aw. The idea of using linear value functions for dynamic programming was proposed, initially, by Bellman et al. 1963] and has been further explored recently [Tsitsiklis and Van Roy, 1996; Koller and Parr, 1999; 2000; Guestrin et al. 2001] . The basic idea is as follows: in the solution algorithms, whether value iteration or policy iteration, we use only value functions within H. Whenever the algorithm takes a step that results in a value function V that is outside this space, we project the result back into the space by finding ....

[Article contains additional citation context not shown here]

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI-01), Seattle, Washington, August 2001. Morgan Kaufmann.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 673--682, 2001.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, pages 673-- 682, 2001.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, pages 673--682, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 673--682, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. Proc. IJCAI-01, pp.673--680, Seattle, WA, 2001.

No context found.

Guestrin, C.; Koller, D.; and Parr, R. 2001. Max-norm projections for factored MDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, 673--682.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, pages 673-- 682, 2001.

No context found.

Guestrin, C.; Koller, D.; and Parr, R. 2001. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, 673--682.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, pages 673--682, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 673--682, 2001.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In IJCAI, pages 673--680, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored mdps. In Proc. of IJCAI, pages 673--680, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In Proc. 17th Intl. Joint Conf. on AI, pp.673--680, Seattle, 2001.

No context found.

Carlos Guestrin, Daphne Koller, and Ronald Parr. Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 673--682, 2001.

No context found.

C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored MDPs. In AAAI, pages 673--679, 2001.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC