Results 1 -
6 of
6
A Comprehensive Survey of Trends in Oracles for Software Testing
, 2013
"... Testing involves examining the behaviour of a system in order to discover potential faults. Determining the desired correct behaviour for a given input is called the “oracle problem”. Oracle automation is important to remove a current bottleneck which inhibits greater overall test automation; witho ..."
Abstract
-
Cited by 20 (4 self)
- Add to MetaCart
Testing involves examining the behaviour of a system in order to discover potential faults. Determining the desired correct behaviour for a given input is called the “oracle problem”. Oracle automation is important to remove a current bottleneck which inhibits greater overall test automation; without oracle automation, the human has to determine whether observed behaviour is correct. The literature on oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of oracle, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the oracle problem and an analysis of trends in this important area of software testing research and practice.
Sigma*: Symbolic Learning of Input-Output Specifications
"... We present Sigma ∗ , a novel technique for learning symbolic models of software behavior. Sigma ∗ addresses the challenge of synthesizing models of software by using symbolic conjectures and abstraction. By combining dynamic symbolic execution to discover symbolic input-output steps of the programs ..."
Abstract
-
Cited by 5 (0 self)
- Add to MetaCart
(Show Context)
We present Sigma ∗ , a novel technique for learning symbolic models of software behavior. Sigma ∗ addresses the challenge of synthesizing models of software by using symbolic conjectures and abstraction. By combining dynamic symbolic execution to discover symbolic input-output steps of the programs and counterexample guided abstraction refinement to over-approximate program behavior, Sigma ∗ transforms arbitrary source representation of programs into faithful input-output models. We define a class of stream filters—programs that process streams of data items—for which Sigma ∗ converges to a complete model if abstraction refinement eventually builds up a sufficiently strong abstraction. In other words, Sigma ∗ is complete relative to abstraction. To represent inferred symbolic models, we use a variant of symbolic transducers that can be effectively composed and equivalence checked. Thus, Sigma ∗ enables fully automatic analysis of behavioral properties such as commutativity, reversibility and idempotence, which is useful for web sanitizer verification and stream programs compiler optimizations, as we show experimentally. We also show how models inferred by Sigma ∗ can boost performance of stream programs by parallelized code generation.
Automated Framework for Formal Operator Task Analysis
"... Aberrant behavior of human operators in safety critical systems can lead to severe or even fatal consequences. Human operators are unique in their decision making capability, judgment and nondeterminism. There is a need for a generalized framework that can allow capturing, modeling and analyzing the ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Aberrant behavior of human operators in safety critical systems can lead to severe or even fatal consequences. Human operators are unique in their decision making capability, judgment and nondeterminism. There is a need for a generalized framework that can allow capturing, modeling and analyzing the interactions between computer systems and human operators where the operators are allowed to deviate from their prescribed behaviors for executing a task. This will provide a formal understanding of the robustness of a computer system against possible aberrant behaviors by its human operators. We provide a framework for (i) modeling the human operators and the computer systems; (ii) formulating tolerable human operator action variations(protection envelope); (iii) determining whether the computer system can maintain its guarantees if the human operators operate within their protection envelopes; and finally, (iv) determining robustness of the computer system under weakening of the protection envelopes. We present Tutela, a tool that assists in accomplishing the first and second step, automates the third step and modestly assists in accomplishing the fourth step.
Black-Box Composition: a Dynamic Approach
"... A core issue in component-based software engineering is rep-resented by automatic component adaptation and composi-tion. At behavioural level, an adaptor is an appropriate en-vironment for two or more software components to correctly interoperate with respect to a certain desired behavioural propert ..."
Abstract
- Add to MetaCart
(Show Context)
A core issue in component-based software engineering is rep-resented by automatic component adaptation and composi-tion. At behavioural level, an adaptor is an appropriate en-vironment for two or more software components to correctly interoperate with respect to a certain desired behavioural property. This, however, requires for the behavioural inter-faces of all components to be well-specified, which is not al-ways the case in industrial practice. What happens if these components are black-boxes: incomplete specification, no provided models and no source code to extract interfaces from? How could we integrate them? Our approach inter-leaves online monitoring, verification-driven execution and model refinement in order to infer models of the black-box components, provide early access to a part of the system functionality whenever possible and synthesize permissive adaptors. We present both a centralized and a distributed technique, the last one directed towards the exploration and control of remote components.
Sigma*: Symbolic Learning of Stream Filters
"... We present Sigma ∗ , a novel technique for learning symbolic models of software behavior. Sigma ∗ addresses the challenge of synthesizing models of software by using symbolic conjectures and abstraction. By combining dynamic symbolic execution to discover symbolic input-output steps of the programs ..."
Abstract
- Add to MetaCart
(Show Context)
We present Sigma ∗ , a novel technique for learning symbolic models of software behavior. Sigma ∗ addresses the challenge of synthesizing models of software by using symbolic conjectures and abstraction. By combining dynamic symbolic execution to discover symbolic input-output steps of the programs and counterexample guided abstraction refinement to over-approximate program behavior, Sigma ∗ transforms arbitrary source representation of programs into faithful input-output models. We define a class of stream filters—programs that process streams of data items—for which Sigma ∗ converges to a complete model if abstraction refinement eventually builds up a sufficiently strong abstraction. In other words, Sigma ∗ is complete relative to abstraction. To represent inferred symbolic models, we use a variant of symbolic transducers that can be effectively composed and equivalence checked. Thus, Sigma ∗ enables fully automatic analysis of behavioral properties such as commutativity, reversibility and idempotence, which is useful for web sanitizer verification and stream programs compiler optimizations, as we show experimentally. We also show how models inferred by Sigma ∗ can be used to boost performance of stream programs through parallelized code generation. 1.
1The Oracle Problem in Software Testing: A Survey
"... Abstract—Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automati ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice. Index Terms—Test oracle; Automatic testing; Testing formalism. F
