#### DMCA

## Automated verification and strategy synthesis for probabilistic systems

Citations: | 7 - 1 self |

### Citations

5610 | Reinforcement Learning: An Introduction
- Sutton, Barto
- 1998
(Show Context)
Citation Context ...satisfies a property φ, or which optimises a specified objective. This is more in line with the way that MDPs are used in other fields, such as planning under uncertainty [34], reinforcement learning =-=[42]-=- or optimal control [8]. In the context of probabilistic model checking, the strategy synthesis problem has generally received less attention than the dual problem of verification, despite being solve... |

3935 |
Dynamic Programming
- Bellman
- 1957
(Show Context)
Citation Context ...rlier, an extended version of this paper, available from [49], presents full details of these methods. Coverage of this material can also be found in, for example, [21,5,2] and standard texts on MDPs =-=[6,26,38]-=-. 3.3 Strategy Synthesis for Probabilistic Reachability To synthesise optimal strategies for probabilistic reachability, it suffices to consider memoryless deterministic strategies. For this class of ... |

1900 |
Markov Decision Processes: Discrete Stochastic Dynamic Programming
- Puterman
- 1994
(Show Context)
Citation Context ...rlier, an extended version of this paper, available from [49], presents full details of these methods. Coverage of this material can also be found in, for example, [21,5,2] and standard texts on MDPs =-=[6,26,38]-=-. 3.3 Strategy Synthesis for Probabilistic Reachability To synthesise optimal strategies for probabilistic reachability, it suffices to consider memoryless deterministic strategies. For this class of ... |

1125 |
Finite Markov Chains,
- Kemeny, Snell
- 1976
(Show Context)
Citation Context ...r of M is fully probabilistic. Formally, we can represent this using an (infinite) induced discrete-time Markov chain, whose states are finite paths of M. This leads us, using a standard construction =-=[28]-=-, to the definition of a probability measure PrσM,s over infinite paths IPathM,s, capturing the behaviour of M from state s under strategy σ. We will also use, for a random variable X : IPathM,s → R>0... |

740 |
Dynamic Programming and Markov processes
- Howard
- 1960
(Show Context)
Citation Context ...rlier, an extended version of this paper, available from [49], presents full details of these methods. Coverage of this material can also be found in, for example, [21,5,2] and standard texts on MDPs =-=[6,26,38]-=-. 3.3 Strategy Synthesis for Probabilistic Reachability To synthesise optimal strategies for probabilistic reachability, it suffices to consider memoryless deterministic strategies. For this class of ... |

662 | Uppaal in a Nutshell.
- Larsen, Pettersson, et al.
- 1997
(Show Context)
Citation Context ...ed methods for generating the automata needed to synthesise strategies for temporal logics such as LTL. Parallels can also be drawn with verification techniques for timed systems: for example, UPPAAL =-=[33]-=-, a model checker developed for verifying timed automata, has been used to great success for synthesising solutions to real-time task scheduling problems, and is in many cases superior to alternative ... |

315 | Reasoning about infinite computations.
- Vardi, Wolper
- 1994
(Show Context)
Citation Context ...er the computation of maximally optimal probabilities. A DRA A with alphabet α represents a set of infinite words L(A) ⊆ αω. For any LTL formula ψ using atomic propositions from AP , we can construct =-=[45,18,5]-=- a DRA Aψ with alphabet 2AP that represents it, i.e., such that an infinite path pi = s0 a0−→s1 a1−→s2 . . . ofM satisfies ψ if and only if Lab(s0)Lab(s1)Lab(s2) . . . is in L(Aψ). We then proceed by ... |

236 | Prism 4.0: Verification of probabilistic real-time systems
- Kwiatkowska, Norman, et al.
- 2011
(Show Context)
Citation Context ...ormance properties, for example, “the maximum probability of the airbag failing to deploy within 0.02 seconds is at most 10−6”. Tool support, in the form of probabilistic model checkers such as PRISM =-=[29]-=- and MRMC [27], has been used to verify quantitative properties of a wide variety of real-life systems, from wireless communication protocols [19], to aerospace designs [9], to DNA circuits [32]. One ... |

206 | The complexity of stochastic games.
- Condon
- 1992
(Show Context)
Citation Context ... For reachability properties (i.e., 〈〈C〉〉P./ p[ F b ]), memoryless deterministic strategies suffice and optimal values and strategies can be computed either with value iteration or strategy iteration =-=[16,17]-=-. For an LTL property ψ, we again reduce 14 Marta Kwiatkowska and David Parker s0ss4ss3spseastss1ssouths0.8s0.1s{goal1}ss2ss5s{hazard}s0.1s{goal2}s{goal2}ssouths1-ps0.6 0.4sstuckseastsstucks0.4s0.6 we... |

187 |
NeuroDynamic Programming (Optimization and
- Bertsekas, Tsitsiklis
- 1996
(Show Context)
Citation Context ...or which optimises a specified objective. This is more in line with the way that MDPs are used in other fields, such as planning under uncertainty [34], reinforcement learning [42] or optimal control =-=[8]-=-. In the context of probabilistic model checking, the strategy synthesis problem has generally received less attention than the dual problem of verification, despite being solved in essentially the sa... |

169 |
Formal verification of probabilistic systems.
- Alfaro
- 1997
(Show Context)
Citation Context ...ynthesised strategy. As mentioned earlier, an extended version of this paper, available from [49], presents full details of these methods. Coverage of this material can also be found in, for example, =-=[21,5,2]-=- and standard texts on MDPs [6,26,38]. 3.3 Strategy Synthesis for Probabilistic Reachability To synthesise optimal strategies for probabilistic reachability, it suffices to consider memoryless determi... |

134 | Policy optimization for dynamic power management,
- Benini, Paleologo, et al.
- 1999
(Show Context)
Citation Context ...i) Dynamic power management. The problem of synthesising optimal (randomised) control strategies to switch between power states in electronic devices can be solved using optimisation problems on MDPs =-=[7]-=- or, alternatively, with multi-objective strategy synthesis for MDPs [21]. In application domains such as these, probabilistic model checking offers various benefits. Firstly, as mentioned above, temp... |

120 |
Interactive Markov Chains: The Quest for Quantified Quality,
- Hermanns
- 2002
(Show Context)
Citation Context ...Verification techniques have also been developed for models that incorporate both nondeterminism and continuous notions of time, including probabilistic timed automata [35], interactive Markov chains =-=[25]-=- and Markov automata [43]. Similarly, progress is being made on verification techniques for models with continuous state spaces, and hybrid models that mix both discrete and continuous elements [48,44... |

96 |
J.P.: Principles of Model Checking.
- Baier, Katoen
- 2008
(Show Context)
Citation Context ...ynthesised strategy. As mentioned earlier, an extended version of this paper, available from [49], presents full details of these methods. Coverage of this material can also be found in, for example, =-=[21,5,2]-=- and standard texts on MDPs [6,26,38]. 3.3 Strategy Synthesis for Probabilistic Reachability To synthesise optimal strategies for probabilistic reachability, it suffices to consider memoryless determi... |

91 | Exploiting structure to efficiently solve large scale Partially Observable Markov Decision Processes.
- Poupart
- 2005
(Show Context)
Citation Context ... Although fundamental decision problems are undecidable in the context of partial observability [3], practical implementations have been developed for a few cases [11,24] and some tool support exists =-=[36]-=-. Developing efficient methods for useful problem classes is an important challenge. – Robustness and uncertainty. In many potential applications of strategy synthesis, such as the generation of contr... |

84 | On algorithms for simple stochastic games,
- Condon
- 1993
(Show Context)
Citation Context ... For reachability properties (i.e., 〈〈C〉〉P./ p[ F b ]), memoryless deterministic strategies suffice and optimal values and strategies can be computed either with value iteration or strategy iteration =-=[16,17]-=-. For an LTL property ψ, we again reduce 14 Marta Kwiatkowska and David Parker s0ss4ss3spseastss1ssouths0.8s0.1s{goal1}ss2ss5s{hazard}s0.1s{goal2}s{goal2}ssouths1-ps0.6 0.4sstuckseastsstucks0.4s0.6 we... |

81 | Symbolic Model Checking for Probabilistic Timed Automata
- Kwiatkowska, Norman, et al.
- 2007
(Show Context)
Citation Context ...rete-time probabilistic models. Verification techniques have also been developed for models that incorporate both nondeterminism and continuous notions of time, including probabilistic timed automata =-=[35]-=-, interactive Markov chains [25] and Markov automata [43]. Similarly, progress is being made on verification techniques for models with continuous state spaces, and hybrid models that mix both discret... |

79 |
Improved automata generation for linear temporal logic. In:
- Daniele, Giunchiglia, et al.
- 1999
(Show Context)
Citation Context ...er the computation of maximally optimal probabilities. A DRA A with alphabet α represents a set of infinite words L(A) ⊆ αω. For any LTL formula ψ using atomic propositions from AP , we can construct =-=[45,18,5]-=- a DRA Aψ with alphabet 2AP that represents it, i.e., such that an infinite path pi = s0 a0−→s1 a1−→s2 . . . ofM satisfies ψ if and only if Lab(s0)Lab(s1)Lab(s2) . . . is in L(Aψ). We then proceed by ... |

75 | The Ins and Outs of the Probabilistic Model Checker MRMC.
- Kaoten, Zapreev, et al.
- 2009
(Show Context)
Citation Context ...ties, for example, “the maximum probability of the airbag failing to deploy within 0.02 seconds is at most 10−6”. Tool support, in the form of probabilistic model checkers such as PRISM [29] and MRMC =-=[27]-=-, has been used to verify quantitative properties of a wide variety of real-life systems, from wireless communication protocols [19], to aerospace designs [9], to DNA circuits [32]. One of the key str... |

50 |
Markovian decision processes with uncertain transition probabilities.
- Satia, Lave
- 1973
(Show Context)
Citation Context ...d the original value of 0.5, i.e., we 2 This notion can be captured more cleanly by annotating transitions directly with probability intervals [40], or with more general specifications of uncertainty =-=[39]-=-. Here, we just aim to give a simple illustration of using a stochastic 2-player game. Automated Verification and Strategy Synthesis for Probabilistic Systems 15 let p = 0.5−∆, q = 0.5+∆ and vary ∆. F... |

46 | A formal analysis of Bluetooth device discovery
- DUFLOT, KWIATKOWSKA, et al.
- 2006
(Show Context)
Citation Context ...he form of probabilistic model checkers such as PRISM [29] and MRMC [27], has been used to verify quantitative properties of a wide variety of real-life systems, from wireless communication protocols =-=[19]-=-, to aerospace designs [9], to DNA circuits [32]. One of the key strengths of probabilistic model checking, in contrast to, for example, approximate analysis techniques based on Monte Carlo simulation... |

42 | On decision problems for probabilistic Büchi automata.
- Baier, Bertrand, et al.
- 2008
(Show Context)
Citation Context ...this is unrealistic, which could lead to strategies being synthesised that are not feasible in practice. Although fundamental decision problems are undecidable in the context of partial observability =-=[3]-=-, practical implementations have been developed for a few cases [11,24] and some tool support exists [36]. Developing efficient methods for useful problem classes is an important challenge. – Robustne... |

41 | Multiobjective model checking of Markov decision processes.
- Etessami, Kwiatkowska, et al.
- 2008
(Show Context)
Citation Context ... . . ∧ P./npn [ψn ] of probabilistic LTL properties. For MDP M and strategy σ, M, σ |=φ if M, σ |= P./1p1 [ψ1 ] for all 1 6 i 6 n. An algorithm for multi-objective LTL strategy synthesis was given in =-=[20]-=-, although here we describe an adapted version, based on [22], using deterministic Rabin automata. The overall approach is similar to standard (single-objective) LTL strategy synthesis in that it cons... |

40 | Automated verification techniques for probabilistic systems.
- Forejt, Kwiatkowska, et al.
- 2011
(Show Context)
Citation Context ...ised) control strategies to switch between power states in electronic devices can be solved using optimisation problems on MDPs [7] or, alternatively, with multi-objective strategy synthesis for MDPs =-=[21]-=-. In application domains such as these, probabilistic model checking offers various benefits. Firstly, as mentioned above, temporal logics provide an expressive means of formally specifying the goals ... |

35 | Receding horizon temporal logic planning
- Wongpiromsarn, Topcu, et al.
- 2012
(Show Context)
Citation Context ...on to date include: (i) Robotics. In recent years, temporal logics such as LTL have grown increasingly popular as a means to specify tasks when synthesising controllers for robots or embedded systems =-=[47]-=-. In the presence of uncertainty, e.g. due to unreliable sensors or actuators, optimal controller synthesis can be performed using MDP model checking techniques [31]. (ii) Security. In the context of ... |

32 | Controller synthesis for probabilistic systems.
- Baier, Großer, et al.
- 2004
(Show Context)
Citation Context ...or PCTL* for verification of MDPs. This is because our primary focus in this tutorial is not verification, but strategy synthesis, for which the treatment of branching-time logics is more challenging =-=[4,10]-=-. For an MDP M, state s and strategy σ of M, and property φ, we write M, s, σ |=φ to denote that, when starting from s, and operating under σ, M satisfies φ. Generally, we are interested in the behavi... |

32 | Motion planning and control from temporal logic specifications with probabilistic satisfaction guarantees.
- Lahijanian, Wasniewski, et al.
- 2010
(Show Context)
Citation Context ...ers for robots or embedded systems [47]. In the presence of uncertainty, e.g. due to unreliable sensors or actuators, optimal controller synthesis can be performed using MDP model checking techniques =-=[31]-=-. (ii) Security. In the context of computer security, model checking has been used to synthesise strategies for malicious attackers, which represent flaws in security systems or protocols. Probability... |

28 | Design and analysis of DNA strand displacement devices using probabilistic model checking.
- Lakin, Parker, et al.
- 2012
(Show Context)
Citation Context ...PRISM [29] and MRMC [27], has been used to verify quantitative properties of a wide variety of real-life systems, from wireless communication protocols [19], to aerospace designs [9], to DNA circuits =-=[32]-=-. One of the key strengths of probabilistic model checking, in contrast to, for example, approximate analysis techniques based on Monte Carlo simulation, is 2 Marta Kwiatkowska and David Parker the ab... |

27 | Task Graph Scheduling using Timed Automata
- Abdeddaim, Kerbaa, et al.
- 2003
(Show Context)
Citation Context ...ed for verifying timed automata, has been used to great success for synthesising solutions to real-time task scheduling problems, and is in many cases superior to alternative state-of-the-art methods =-=[1]-=-. In this paper, we give an overview of methods for performing verification and strategy synthesis on probabilistic systems. Our focus is primarily on algorithmic issues: we introduce the basic ideas,... |

27 | Quantitative synthesis for concurrent programs
- Cerný, Chatterjee, et al.
- 2011
(Show Context)
Citation Context ...ed that are not feasible in practice. Although fundamental decision problems are undecidable in the context of partial observability [3], practical implementations have been developed for a few cases =-=[11,24]-=- and some tool support exists [36]. Developing efficient methods for useful problem classes is an important challenge. – Robustness and uncertainty. In many potential applications of strategy synthesi... |

26 | Model-checking Markov chains in the presence of uncertainties
- Sen, Viswanathan, et al.
- 2006
(Show Context)
Citation Context ...ute this value for various intervals [p, q] centred around the original value of 0.5, i.e., we 2 This notion can be captured more cleanly by annotating transitions directly with probability intervals =-=[40]-=-, or with more general specifications of uncertainty [39]. Here, we just aim to give a simple illustration of using a stochastic 2-player game. Automated Verification and Strategy Synthesis for Probab... |

24 | Quantitative multiobjective verification for probabilistic systems
- Forejt, Kwiatkowska, et al.
- 2011
(Show Context)
Citation Context ...M and strategy σ, M, σ |=φ if M, σ |= P./1p1 [ψ1 ] for all 1 6 i 6 n. An algorithm for multi-objective LTL strategy synthesis was given in [20], although here we describe an adapted version, based on =-=[22]-=-, using deterministic Rabin automata. The overall approach is similar to standard (single-objective) LTL strategy synthesis in that it constructs a product automaton and reduces the problem to (multi-... |

19 | The COMPASS Approach: Correctness, modelling and performability of aerospace systems,”
- Bozzano, Cimatti, et al.
- 2009
(Show Context)
Citation Context ...del checkers such as PRISM [29] and MRMC [27], has been used to verify quantitative properties of a wide variety of real-life systems, from wireless communication protocols [19], to aerospace designs =-=[9]-=-, to DNA circuits [32]. One of the key strengths of probabilistic model checking, in contrast to, for example, approximate analysis techniques based on Monte Carlo simulation, is 2 Marta Kwiatkowska a... |

18 |
Safety verification for probabilistic hybrid systems.
- Zhang, She, et al.
- 2012
(Show Context)
Citation Context ...s [25] and Markov automata [43]. Similarly, progress is being made on verification techniques for models with continuous state spaces, and hybrid models that mix both discrete and continuous elements =-=[48,44]-=-. Developing efficient strategy synthesis techniques for such models will bring the benefits of the methods discussed in this paper to a much wider range of application domains. Acknowledgments. The a... |

17 | Automatic verification of competitive stochastic systems
- Chen, Forejt, et al.
- 2012
(Show Context)
Citation Context ...synthesis on SMGs, we generate strategies either for an individual player, or for a coalition C ⊆ Π of players. We extend the definition of properties given in Defn. 4 in the style of the logic rPATL =-=[14]-=-. Definition 10 (Multi-player strategy synthesis). For a property P./ p[ψ ] or Rr./ x[ ρ ] and a coalition C ⊆ Π of players, (zero-sum) multi-player strategy synthesis is expressed by a query 〈〈C〉〉P./... |

14 |
Stochastic games with branching-time winning objectives
- Brázdil, Brožek, et al.
- 2006
(Show Context)
Citation Context ...or PCTL* for verification of MDPs. This is because our primary focus in this tutorial is not verification, but strategy synthesis, for which the treatment of branching-time logics is more challenging =-=[4,10]-=-. For an MDP M, state s and strategy σ of M, and property φ, we write M, s, σ |=φ to denote that, when starting from s, and operating under σ, M satisfies φ. Generally, we are interested in the behavi... |

14 | PRISM-games: A model checker for stochastic multi-player games
- Chen, Forejt, et al.
(Show Context)
Citation Context ...ails of the algorithms needed to perform strategy synthesis and additional worked examples, is available as [30]. The examples in both versions of the paper can be run using PRISM (and its extensions =-=[15]-=-). Accompanying PRISM files are available online [49]. 2 Markov Decision Processes In the majority of this paper, we focus on Markov decision processes (MDPs), which model systems that exhibit both pr... |

12 |
Formal analysis of pin block attacks
- Steel
(Show Context)
Citation Context ...hecking has been used to synthesise strategies for malicious attackers, which represent flaws in security systems or protocols. Probability is also often a key ingredient of security; for example, in =-=[41]-=-, probabilistic model checking of MDPs was used to generate PIN guessing attacks against hardware security modules. (iii) Dynamic power management. The problem of synthesising optimal (randomised) con... |

10 | Pareto curves for probabilistic model checking
- Forejt, Kwiatkowska, et al.
- 2012
(Show Context)
Citation Context ...to LTL properties. The LP-based approach sketched above has beeen extended [22] to include reward objectives of the form Rr./ x[ C ]. An alternative approach, based on value iteration, rather than LP =-=[23]-=-, allows the addition of stepbounded reward objectives Rr./ x[ C 6k ] (and also provides significant gains in efficiency for both classes of properties). (iii) numerical multi-objective queries genera... |

8 |
Strategy improvement and randomized subexponential algorithms for stochastic parity games
- Chatterjee, Henzinger
- 2006
(Show Context)
Citation Context ...e strategy improvement algorithm of [13] to directly compute probabilities for Rabin objectives, or convert the DRA to a deterministic parity automaton and compute probabilities for parity objectives =-=[12]-=-. Example 6. To give an example of strategy synthesis using stochastic games, we consider a simple extension of the MDP M from the running example (Fig. 1). We assume that the existing choices in the ... |

7 | Efficient modelling and generation of Markov automata
- Timmer, Katoen, et al.
- 2012
(Show Context)
Citation Context ...ave also been developed for models that incorporate both nondeterminism and continuous notions of time, including probabilistic timed automata [35], interactive Markov chains [25] and Markov automata =-=[43]-=-. Similarly, progress is being made on verification techniques for models with continuous state spaces, and hybrid models that mix both discrete and continuous elements [48,44]. Developing efficient s... |

6 | Strategy improvement for stochastic Rabin and Strett Games
- Chatterjee, Henzinger
- 2006
(Show Context)
Citation Context ...pler one on the product of the game G and a deterministic automaton representing ψ. Unlike MDPs, there is no notion of end components. Instead, we can either use the strategy improvement algorithm of =-=[13]-=- to directly compute probabilities for Rabin objectives, or convert the DRA to a deterministic parity automaton and compute probabilities for parity objectives [12]. Example 6. To give an example of s... |

6 | A.: Formula-free Finite Abstractions for Linear Temporal Verification of Stochastic Hybrid Systems. In:
- Tkachev, Abate
- 2013
(Show Context)
Citation Context ...s [25] and Markov automata [43]. Similarly, progress is being made on verification techniques for models with continuous state spaces, and hybrid models that mix both discrete and continuous elements =-=[48,44]-=-. Developing efficient strategy synthesis techniques for such models will bring the benefits of the methods discussed in this paper to a much wider range of application domains. Acknowledgments. The a... |

5 | Polynomial-time verification of PCTL properties of MDPs with convex uncertainties.
- Puggelli, Li, et al.
- 2013
(Show Context)
Citation Context ...e robust in these environments, is important. We gave a very simple illustration of uncertain probabilistic behaviour in Sec. 5. Developing more sophisticated approaches is an active area of research =-=[46,37]-=-. – Continuous time and space. In this paper, we focused on discrete-time probabilistic models. Verification techniques have also been developed for models that incorporate both nondeterminism and con... |

5 | Robust control of uncertain Markov decision processes with temporal logic specifications
- Wolff, Topcu, et al.
(Show Context)
Citation Context ...e robust in these environments, is important. We gave a very simple illustration of uncertain probabilistic behaviour in Sec. 5. Developing more sophisticated approaches is an active area of research =-=[46,37]-=-. – Continuous time and space. In this paper, we focused on discrete-time probabilistic models. Verification techniques have also been developed for models that incorporate both nondeterminism and con... |

4 | M.: Verification of partial-information probabilistic systems using counterexample-guided refinements
- Giro, Rabe
(Show Context)
Citation Context ...ed that are not feasible in practice. Although fundamental decision problems are undecidable in the context of partial observability [3], practical implementations have been developed for a few cases =-=[11,24]-=- and some tool support exists [36]. Developing efficient methods for useful problem classes is an important challenge. – Robustness and uncertainty. In many potential applications of strategy synthesi... |

1 |
A.: Planning with Markov Decision Processes: An AI Perspective
- Masuam
- 2012
(Show Context)
Citation Context ...finds some strategy ofM that satisfies a property φ, or which optimises a specified objective. This is more in line with the way that MDPs are used in other fields, such as planning under uncertainty =-=[34]-=-, reinforcement learning [42] or optimal control [8]. In the context of probabilistic model checking, the strategy synthesis problem has generally received less attention than the dual problem of veri... |