Citation Context

...re arises from introducing the primitive and compound events in the work of Becker et al. (2004). This reward structure is necessary to capture the characteristics of the Mars rover benchmark. Interestingly, this extension does not complicate our proposed solution methods in any way. Note that the stochastic shortest path formulation (right side of Figure 1) inherently eliminates any loops because time always advances when an action is taken. Therefore, every state in that representation may be visited at most once. This property is commonly used when an MDP is formulated as a linear program (Puterman, 2005). The solution of a DEC-MDP is a deterministic stationary policy π = (π1, π2), where πi : Si → Ai is the standard MDP policy (Puterman, 2005) for agent i. In particular, πi(si) represents the action taken by agent i in state si. To define the bilinear program, we use variables x(s1, a1) to denote the probability that agent 1 visits state s1 and takes action a1 and y(s2, a2) to denote the same for agent 2. These are the standard dual variables in MDP formulation. Given a solution in terms of x for agent 1, the policy is calculated for s ∈ S1 as follows, breaking ties arbitrarily. π1(s) = arg m...

Citation Context

...ith a mixed integer linear program (MILP), derived for Eq. (1) as Petrik and Zilberstein (2007b) describe. Although Eq. (1) can also be modeled as a linear complementarity problem (LCP) (Murty, 1988; =-=Cottle et al., 1992-=-), we do not evaluate that option experimentally because LCPs are closely related to MILPs (Rosen, 1986). We expect these two formulations to exhibit similar performance. We also do not compare to any...

Citation Context

...ng certain independence assumptions (Becker, Zilberstein, & Lesser, 2003) or by adding explicit communication actions (Goldman & Zilberstein, 2008). DEC-POMDPs are closely related to extensive games (=-=Rubinstein, 1997-=-). In fact, any DEC-POMDP represents an exponentially larger extensive game with a common objective. Unfortunately, DEC-POMDPs with just two agents are intractable in general, unlike MDPs that can be ...

Citation Context

...out the world–must cooperate with each other in order to achieve some joint objective. Such problems are common in practice and can be modeled as decentralized partially observable MDPs (DEC-POMDPs) (=-=Bernstein, Zilberstein, & Immerman, 2000-=-). Some refinements of this model have been studied, for example by making certain independence assumptions (Becker, Zilberstein, & Lesser, 2003) or by adding explicit communication actions (Goldman &...

Citation Context

...the bilinear term of the objective function are independently constrained. The theory of nonseparable bilinear programs is much more complicated and the corresponding algorithms are not as efficient (=-=Horst & Tuy, 1996-=-). Thus, we limit the discussion in this paper to separable bilinear programs and often omit the term “separable”. As discussed later in more detail, a separable bilinear program may be seen as a conc...

Citation Context

...ram. While the algorithm often performs well in practice, it tends to converge to a suboptimal solution (Mangasarian, 1995). When applied to DEC-MDPs, this algorithm is essentially identical to JESP (=-=Nair, Tambe, Yokoo, Pynadath, & Marsella, 2003-=-)–one of the early solution methods. In the following, we use f(w, x, y, z) to denote the objective value of Eq. (1). The rest of this section presents a new anytime algorithm for solving bilinear pro...

Citation Context

... model to cooperative multiagent problems. One possibility is to assume that all the agents share all the information about the underlying state. This results in a multiagent Markov decision process (=-=Boutilier, 1999-=-), which is essentially an MDP with a factored action set. A more complex alternative is to allow only partial sharing of information among agents. In these settings, several agents–each having differ...

Citation Context

...nal CSA and with a mixed integer linear program (MILP), derived for Eq. (1) as Petrik and Zilberstein (2007b) describe. Although Eq. (1) can also be modeled as a linear complementarity problem (LCP) (=-=Murty, 1988-=-; Cottle et al., 1992), we do not evaluate that option experimentally because LCPs are closely related to MILPs (Rosen, 1986). We expect these two formulations to exhibit similar performance. We also ...

Citation Context

...d through a common reward function that depends on their states. The coverage set algorithm (CSA) was the first optimal algorithm to solve efficiently transition and observation independent DEC-MDPs (=-=Becker, Zilberstein, Lesser, & Goldman, 2004-=-). By exploiting the fact that the interaction between the agents is limited compared to their individual local problems, CSA can solve problems that cannot be solved by the more general exact DEC-POM...

Citation Context

...blems (Becker, c○2009 AI Access Foundation. All rights reserved.Petrik & Zilberstein Lesser, & Zilberstein, 2004; Kim, Nair, Varakantham, Tambe, & Yokoo, 2006) or provide only approximate solutions (=-=Emery-Montemerlo, Gordon, Schneider, & Thrun, 2004-=-; Nair, Roth, Yokoo, & Tambe, 2004; Seuken & Zilberstein, 2007). In this paper, we introduce an efficient algorithm for several restricted classes, most notably decentralized MDPs with transition and ...

Citation Context

...the intersection points of planes defined by the current solutions in ˜ X. That guarantees that g(y) is eventually known precisely (Becker et al., 2004). A similar approach was also taken for POMDPs (=-=Cheng, 1988-=-). The upper bound on the number of intersection points in CSA is ( | ˜ X| ) dim Y . The principal problem is that the bound is exponential in the dimension of Y , and experiments do not show a slower...

Citation Context

...entralized partially observable MDPs (DEC-POMDPs) (Bernstein, Zilberstein, & Immerman, 2000). Some refinements of this model have been studied, for example by making certain independence assumptions (=-=Becker, Zilberstein, & Lesser, 2003-=-) or by adding explicit communication actions (Goldman & Zilberstein, 2008). DEC-POMDPs are closely related to extensive games (Rubinstein, 1997). In fact, any DEC-POMDP represents an exponentially la...

Citation Context

...ntractable in general, unlike MDPs that can be solved in polynomial time. Despite recent progress in solving DEC-POMDPs, even state-of-the-art algorithms are generally limited to very small problems (=-=Seuken & Zilberstein, 2008-=-). This has motivated the development of algorithms that either solve a restricted class of problems (Becker, c○2009 AI Access Foundation. All rights reserved.Petrik & Zilberstein Lesser, & Zilberste...

Citation Context

...izon), to handle interdependent observations, and to find Nash equilibria in competitive settings. 2.1 DEC-MDPs As mentioned previously, any transition-independent and observation-independent DECMDP (=-=Becker et al., 2004-=-) may be formulated as a bilinear program. Intuitively, a DECMDP is transition independent when no agent can influence the other agents’ transitions. A DEC-MDP is observation independent when no agent...

Citation Context

...orithm. For this purpose we use the Mars rover problem described earlier. We compared our algorithm with the original CSA and with a mixed integer linear program (MILP), derived for Eq. (1) as Petrik and Zilberstein (2007b) describe. Although Eq. (1) can also be modeled as a linear complementarity problem (LCP) (Murty, 1988; Cottle et al., 1992), we do not evaluate that option experimentally because LCPs are closely related to MILPs (Rosen, 1986). We expect these two formulations to exhibit similar performance. We also do not compare to any of the methods described by Horst and Tuy (1996) and Bennett and Mangasarian (1992) due to their very different nature and high complexity, and because some of these algorithms do not provide any optimality guarantees. In our experiments, we applied the algorithm to randomly generated problem instances with the same parameters that Becker et al. (2003, 2004) used. Each problem instance includes 2 rovers and 6 sites. At each site, the rovers can decide to perform an experiment or to skip the site. Performing experiments takes some time, and all the experiments must be performed in 15 time units. The time required to perform an experiment is drawn from a discrete normal distri...

Citation Context

...ts reserved.Petrik & Zilberstein Lesser, & Zilberstein, 2004; Kim, Nair, Varakantham, Tambe, & Yokoo, 2006) or provide only approximate solutions (Emery-Montemerlo, Gordon, Schneider, & Thrun, 2004; =-=Nair, Roth, Yokoo, & Tambe, 2004-=-; Seuken & Zilberstein, 2007). In this paper, we introduce an efficient algorithm for several restricted classes, most notably decentralized MDPs with transition and observation independence (Becker e...

Citation Context

...rik & Zilberstein, 2007b). This is particularly useful for infinite-horizon DEC-MDPs. For example, consider the infinitehorizon version of the Multiple Access Broadcast Channel (MABC) (Rosberg, 1983; =-=Ooi & Wornell, 1996-=-). In this problem, which has been used widely in recent studies of decentralized decision making, two communication devices share a single channel, and they need to periodically transmit some data. H...

Citation Context

...ipulation (Pang, Trinkle, & Lo, 1996), bilinear separation (Bennett & Mangasarian, 236A Bilinear Programming Approach for Multiagent Planning 1992), and even general linear complementarity problems (=-=Mangasarian, 1995-=-). We focus on multiagent planning problems where this formulation turns out to be particularly effective. Definition 1. A separable bilinear program in the normal form is defined as follows: maximize...

Citation Context

...inds of separable bilinear problems. When the algorithm is applied to DEC-MDPs, it improves efficiency by several orders of magnitude compared with previous state-of the art algorithms (Becker, 2006; =-=Petrik & Zilberstein, 2007-=-a). In addition, the algorithm provides useful runtime bounds on the approximation error, which makes it more useful as an anytime algorithm. Finally, the algorithm is formulated for general separable...

Citation Context

...s motivated the development of algorithms that either solve a restricted class of problems (Becker, c○2009 AI Access Foundation. All rights reserved.Petrik & Zilberstein Lesser, & Zilberstein, 2004; =-=Kim, Nair, Varakantham, Tambe, & Yokoo, 2006-=-) or provide only approximate solutions (Emery-Montemerlo, Gordon, Schneider, & Thrun, 2004; Nair, Roth, Yokoo, & Tambe, 2004; Seuken & Zilberstein, 2007). In this paper, we introduce an efficient alg...

Citation Context

...tiagent planning problems that can be formulated as such. In addition to multiagent planning problems, bilinear programs can be used to solve a variety of other problems such as robotic manipulation (=-=Pang, Trinkle, & Lo, 1996-=-), bilinear separation (Bennett & Mangasarian, 236A Bilinear Programming Approach for Multiagent Planning 1992), and even general linear complementarity problems (Mangasarian, 1995). We focus on mult...

Citation Context

... problems can be treated within this framework. Besides multiagent coordination problems, bilinear programs have been previously used to solve problems in operations research and global optimization (=-=Sherali & Shetty, 1980-=-; White, 1992; Gabriel, Garca-Bertrand, Sahakij, & Conejo, 2005). Global optimization deals with finding the optimal solutions to problems with multi-extremal objective function. Solution techniques o...

Citation Context

...inequalities: maximize x,y x T Cy subject to A1x ≤ b1 x ≥ 0 A2y ≤ b2 y ≥ 0 (2) The latter formulation can be easily transformed into the normal form using standard transformations of linear programs (=-=Vanderbei, 2001-=-). In particular, we can introduce slack 1. It is possible to define the dimensionality in terms of x, or the minimum of dimensions of x and y. The issue is discussed in Appendix B. 237Petrik & Zilbe...

Citation Context

...man, 2000). Some refinements of this model have been studied, for example by making certain independence assumptions (Becker, Zilberstein, & Lesser, 2003) or by adding explicit communication actions (=-=Goldman & Zilberstein, 2008-=-). DEC-POMDPs are closely related to extensive games (Rubinstein, 1997). In fact, any DEC-POMDP represents an exponentially larger extensive game with a common objective. Unfortunately, DEC-POMDPs wit...

Citation Context

...inds of separable bilinear problems. When the algorithm is applied to DEC-MDPs, it improves efficiency by several orders of magnitude compared with previous state-of the art algorithms (Becker, 2006; =-=Petrik & Zilberstein, 2007-=-a). In addition, the algorithm provides useful runtime bounds on the approximation error, which makes it more useful as an anytime algorithm. Finally, the algorithm is formulated for general separable...

Citation Context

...ar program (Petrik & Zilberstein, 2007b). This is particularly useful for infinite-horizon DEC-MDPs. For example, consider the infinitehorizon version of the Multiple Access Broadcast Channel (MABC) (=-=Rosberg, 1983-=-; Ooi & Wornell, 1996). In this problem, which has been used widely in recent studies of decentralized decision making, two communication devices share a single channel, and they need to periodically ...

Citation Context

...Lesser, & Zilberstein, 2004; Kim, Nair, Varakantham, Tambe, & Yokoo, 2006) or provide only approximate solutions (Emery-Montemerlo, Gordon, Schneider, & Thrun, 2004; Nair, Roth, Yokoo, & Tambe, 2004; =-=Seuken & Zilberstein, 2007-=-). In this paper, we introduce an efficient algorithm for several restricted classes, most notably decentralized MDPs with transition and observation independence (Becker et al., 2003). For the sake o...

Citation Context

...pproach, it is possible to reduce the dimensionality of such problems automatically, without extensive modeling effort. This makes it easy to apply our new method in practice. When applied to DEC-MDPs, the algorithm is much faster than the existing CSA method, on average reducing computation time by four orders of magnitude. We also show that a variety of other coordination problems can be treated within this framework. Besides multiagent coordination problems, bilinear programs have been previously used to solve problems in operations research and global optimization (Sherali & Shetty, 1980; White, 1992; Gabriel, Garca-Bertrand, Sahakij, & Conejo, 2005). Global optimization deals with finding the optimal solutions to problems with multi-extremal objective function. Solution techniques often share the same idea and are based on cutting plane methods. The main idea is to iteratively restrict the set of feasible solutions, while improving the incumbent 266 A Bilinear Programming Approach for Multiagent Planning solution. Horst and Tuy (1996) provide an excellent overview of these techniques. These algorithms have different characteristics and cannot be directly compared to the algorithm we deve...

Citation Context

...olving these kinds of separable bilinear problems. When the algorithm is applied to DEC-MDPs, it improves efficiency by several orders of magnitude compared with previous state-of the art algorithms (=-=Becker, 2006-=-; Petrik & Zilberstein, 2007a). In addition, the algorithm provides useful runtime bounds on the approximation error, which makes it more useful as an anytime algorithm. Finally, the algorithm is form...

Citation Context

...ough Eq. (1) can also be modeled as a linear complementarity problem (LCP) (Murty, 1988; Cottle et al., 1992), we do not evaluate that option experimentally because LCPs are closely related to MILPs (=-=Rosen, 1986-=-). We expect these two formulations to exhibit similar performance. We also do not compare to any of the methods described by Horst and Tuy (1996) and Bennett and Mangasarian (1992) due to their very ...

Citation Context

...tengel, 1994), and integer linear program formulation of DEC-POMDPs (Aras & Charpillet, 2007). The approach we develop is closely related to event-driven DECPOMDPs (Becker et al., 2004), but it is in general more efficient. Nevertheless, the size of the bilinear program is exponential in the size of the DEC-POMDP. This can be expected since solving DEC-POMDPs is NEXP-complete (Bernstein et al., 2000), while solving bilinear programs is NP-complete (Mangasarian, 1995). Because the general formulation in this case is somewhat cumbersome, we only illustrate it using the following simple example. Aras (2008) provides the details of a similar construction. Example 6. Consider the problem depicted in Figure 3, assuming that the agents are cooperative. The actions of the other agent are not observable, as denoted by the information sets. This approach can be generalized to any problem with any observable sets as long as the perfect recall condition is satisfied. Agents satisfy the perfect recall condition when they remember the set of actions taken in the prior moves (Osborne & Rubinstein, 1994). Rewards are only collected in the leaf-nodes in this case. The variables on the edges represent the prob...

Citation Context

...framework. Besides multiagent coordination problems, bilinear programs have been previously used to solve problems in operations research and global optimization (Sherali & Shetty, 1980; White, 1992; =-=Gabriel, Garca-Bertrand, Sahakij, & Conejo, 2005-=-). Global optimization deals with finding the optimal solutions to problems with multi-extremal objective function. Solution techniques often share the same idea and are based on cutting plane methods...

Citation Context

...The approach in this case may be similar to linear complementarity problem formulation of extensive games (Koller, Megiddo, & von Stengel, 1994), and integer linear program formulation of DEC-POMDPs (=-=Aras & Charpillet, 2007-=-). The approach we develop is closely related to event-driven DECPOMDPs (Becker et al., 2004), but it is in general more efficient. Nevertheless, the size of the bilinear program is exponential in the...

Citation Context

...MDP to refer to transition and observation independent DEC-MDP. The DEC-MDP model has proved useful in several multiagent planning domains. One example that we use is the Mars rover planning problem (=-=Bresina, Golden, Smith, & Washington, 1999-=-), first formulated as a DEC-MDP by Becker et al. (2003). This domain involves two autonomous rovers that visit several sites in a given order and may decide to perform certain scientific experiments ...