Results 1 - 10
of
88
Efficient mutation testing by checking invariant violations
- In ISSTA ’09: Proceedings of the 2009 ACM SIGSOFT Int. symposium on Software testing and analysis
, 2009
"... Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a mutation is not detected by the test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the program’s semantics unchanged— a ..."
Abstract
-
Cited by 43 (7 self)
- Add to MetaCart
(Show Context)
Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a mutation is not detected by the test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the program’s semantics unchanged— and thus cannot be detected by any test. Such equivalent mutants have to be eliminated manually, which is tedious. We assess the impact of mutations by checking dynamic invariants. In an evaluation of our JAVALANCHE framework on seven industrial-size programs, we found that mutations that violate invariants are significantly more likely to be detectable by a test suite. As a consequence, mutations with impact on invariants should be focused upon when improving test suites. With less than 3 % of equivalent mutants, our approach provides an efficient, precise, and fully automatic measure of the adequacy of a test suite.
Directed incremental symbolic execution
- In PLDI
, 2011
"... The last few years have seen a resurgence of interest in the use of symbolic execution – a program analysis technique developed more than three decades ago to analyze program execution paths. Scaling symbolic execution and other path-sensitive analysis techniques to large systems remains challenging ..."
Abstract
-
Cited by 30 (8 self)
- Add to MetaCart
(Show Context)
The last few years have seen a resurgence of interest in the use of symbolic execution – a program analysis technique developed more than three decades ago to analyze program execution paths. Scaling symbolic execution and other path-sensitive analysis techniques to large systems remains challenging despite recent algorithmic and technological advances. An alternative to solving the problem of scalability is to reduce the scope of the analysis. One approach that is widely studied in the context of regression analysis is to analyze the differences between two related program versions. While such an approach is intuitive in theory, finding efficient and precise ways to identify program differences, and characterize their effects on how the program executes has proved challenging in practice. In this paper, we present Directed Incremental Symbolic Execution (DiSE), a novel technique for detecting and characterizing the effects of program changes. The novelty of DiSE is to combine the efficiencies of static analysis techniques to compute program difference information with the precision of symbolic execution to explore program execution paths and generate path conditions affected by the differences. DiSE is a complementary technique to other reduction or bounding techniques developed to improve symbolic execution. Furthermore, DiSE does not require analysis results to be carried forward as the software evolves—only the source code for two related program versions is required. A case-study of our implementation of DiSE illustrates its effectiveness at detecting and characterizing the effects of program changes.
Directed test suite augmentation: techniques and tradeoffs
- In FSE’10
, 2010
"... Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. Our preliminary work suggests that several factors influence the cost and effectiveness of test suite augmentation techniques. These inc ..."
Abstract
-
Cited by 27 (4 self)
- Add to MetaCart
(Show Context)
Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. Our preliminary work suggests that several factors influence the cost and effectiveness of test suite augmentation techniques. These include the order in which affected elements are considered while generating test cases, the manner in which existing regression test cases and newly generated test cases are used, and the algorithm used to generate test cases. In this work, we present the results of an empirical study examining these factors, considering two test case generation algorithms (concolic and genetic). The results of our experiment show that the primary factor affecting augmentation is the test case generation algorithm utilized; this affects both cost and effectiveness. The manner in which existing and newly generated test cases are utilized also has a substantial effect on efficiency but a lesser effect on effectiveness. The order in which affected elements are considered turns out to have relatively few effects when using concolic test case generation, but more substantial effects when using genetic test case generation.
ReAssert: Suggesting repairs for broken unit tests,” University of Illinois at Urbana-Champaign
, 2009
"... Abstract—Developers often change software in ways that cause tests to fail. When this occurs, developers must determine whether failures are caused by errors in the code under test or in the test code itself. In the latter case, developers must repair failing tests or remove them from the test suite ..."
Abstract
-
Cited by 24 (4 self)
- Add to MetaCart
(Show Context)
Abstract—Developers often change software in ways that cause tests to fail. When this occurs, developers must determine whether failures are caused by errors in the code under test or in the test code itself. In the latter case, developers must repair failing tests or remove them from the test suite. Reparing tests is time consuming but beneficial, since removing tests reduces a test suite’s ability to detect regressions. Fortunately, simple program transformations can repair many failing tests automatically. We present ReAssert, a novel technique and tool that suggests repairs to failing tests ’ code which cause the tests to pass. Examples include replacing literal values in tests, changing assertion methods, or replacing one assertion with several. If the developer chooses to apply the repairs, ReAssert modifies the code automatically. Our experiments show that ReAssert can repair many common test failures and that its suggested repairs correspond to developers ’ expectations. I.
Directed Test Suite Augmentation
"... Abstract—As software evolves, engineers use regression testing to evaluate its fitness for release. Such testing typically begins with existing test cases, and many techniques have been proposed for reusing these cost-effectively. After reusing test cases, however, it is also important to consider c ..."
Abstract
-
Cited by 19 (6 self)
- Add to MetaCart
(Show Context)
Abstract—As software evolves, engineers use regression testing to evaluate its fitness for release. Such testing typically begins with existing test cases, and many techniques have been proposed for reusing these cost-effectively. After reusing test cases, however, it is also important to consider code or behavior that has not been exercised by existing test cases and generate new test cases to validate these. This process is known as test suite augmentation. In this paper we present a directed test suite augmentation technique, that utilizes results from reuse of existing test cases together with an incremental concolic testing algorithm to augment test suites so that they are coverageadequate for a modified program. We present results of an empirical study examining the effectiveness of our approach. Keywords-regression testing, augmentation, concolic testing I.
Exploiting Program Dependencies for Scalable Multiple-Path Symbolic Execution
"... This paper presents a new technique, called Symbolic Program Decomposition (or SPD), for symbolic execution of multiple paths that is more scalable than existing techniques, which symbolically execute control-flow paths individually. SPD exploits control and data dependencies to avoid analyzing unne ..."
Abstract
-
Cited by 19 (4 self)
- Add to MetaCart
(Show Context)
This paper presents a new technique, called Symbolic Program Decomposition (or SPD), for symbolic execution of multiple paths that is more scalable than existing techniques, which symbolically execute control-flow paths individually. SPD exploits control and data dependencies to avoid analyzing unnecessary combinations of subpaths. SPD can also compute an over-approximation of symbolic execution by abstracting away symbolic subterms arbitrarily, to further scale the analysis at the cost of precision. The paper also presents our implementation and empirical evaluation showing that SPD can achieve savings of orders of magnitude in the pathexploration costs of multiple-path symbolic execution. Finally, the paper presents a study that examines the use of SPD for a particular application: change analysis for test-suite augmentation. Categories and Subject Descriptors: D.2.5 [Software Engineering]: Testing and Debugging—symbolic execution, testing tools;
The Impact of Equivalent Mutants
- SOFTWARE TESTING, VERIFICATION AND VALIDATION WORKSHOPS, 2009. ICSTW '09. INTERNATIONAL CONFERENCE ON
, 2009
"... If a mutation is not killed by a test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the program’s semantics unchanged—and thus cannot be detected by any test. We found such equivalent mutants to be surprisingly common: In an experiment o ..."
Abstract
-
Cited by 18 (3 self)
- Add to MetaCart
(Show Context)
If a mutation is not killed by a test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the program’s semantics unchanged—and thus cannot be detected by any test. We found such equivalent mutants to be surprisingly common: In an experiment on the JAXEN XPATH query engine, 8/20 = 40 % of all mutations turned out to be equivalent. Worse, checking the equivalency took us 15 minutes for a single mutation. Equivalent mutants thus make it impossible to automatically assess test suites by means of mutation testing. To identify equivalent mutants, we are currently investigating the impact of a mutation on the execution: the more a mutation alters the execution, the higher the chance of it being non-equivalent. First experiments assessing the impact on code coverage are promising. 1.
Static Analysis for Probabilistic Programs: Inferring Whole Program Properties from Finitely Many Paths.
"... We propose an approach for the static analysis of probabilistic programs that sense, manipulate, and control based on uncertain data. Examples include programs used in risk analysis, medical decision making and cyber-physical systems. Correctness properties of such programs take the form of queries ..."
Abstract
-
Cited by 16 (1 self)
- Add to MetaCart
(Show Context)
We propose an approach for the static analysis of probabilistic programs that sense, manipulate, and control based on uncertain data. Examples include programs used in risk analysis, medical decision making and cyber-physical systems. Correctness properties of such programs take the form of queries that seek the probabilities of assertions over program variables. We present a static analysis approach that provides guaranteed interval bounds on the values (assertion probabilities) of such queries. First, we observe that for probabilistic programs, it is possible to conclude facts about the behavior of the entire program by choosing a finite, adequate set of its paths. We provide strategies for choosing such a set of paths and verifying its adequacy. The queries are evaluated over each path by a combination of symbolic execution and probabilistic volumebound computations. Each path yields interval bounds that can be summed up with a “coverage ” bound to yield an interval that encloses the probability of assertion for the program as a whole. We demonstrate promising results on a suite of benchmarks from many different sources including robotic manipulators and medical decision making programs.
Test Generation to Expose Changes in Evolving Programs
"... Software constantly undergoes changes throughout its life cycle, and thereby it evolves. As changes are introduced into a code base, we need to make sure that the effect of the changes is thoroughly tested. For this purpose, it is important to generate test cases that can stress the effect of a give ..."
Abstract
-
Cited by 16 (5 self)
- Add to MetaCart
(Show Context)
Software constantly undergoes changes throughout its life cycle, and thereby it evolves. As changes are introduced into a code base, we need to make sure that the effect of the changes is thoroughly tested. For this purpose, it is important to generate test cases that can stress the effect of a given change. In this paper, we propose an automatic test generation solution to this problem. Given a change c, we use dynamic symbolic execution to generate a test input t, which stresses the change. This is done by ensuring (i) the change c is executed by t, and (ii) the effect of c is observable in the output produced by the test t. To construct a change-reaching input, our technique uses distance in control-dependency graph to guide path exploration towards the change. Then, our technique identifies the common programming patterns that may prevent a given change from affecting the program’s output. For each of these patterns we propose methods to tune the change-reaching input into an input that reaches the change and propagates the effect of the change to the output. Our experimental results show that our test generation technique is effective in generating change-exposing inputs for real-world programs.
On Test Generation through Programming in UDITA
"... We present an approach for describing tests using nondeterministic test generation programs. To write test generation programs, we introduce UDITA, a Java-based language with non-deterministic choice operators and an interface for generating linked structures. We also describe new algorithms that ge ..."
Abstract
-
Cited by 15 (6 self)
- Add to MetaCart
We present an approach for describing tests using nondeterministic test generation programs. To write test generation programs, we introduce UDITA, a Java-based language with non-deterministic choice operators and an interface for generating linked structures. We also describe new algorithms that generate concrete tests by efficiently exploring the space of all executions of non-deterministic UDITA programs. We implemented our approach and incorporated it into the official, publicly available repository of Java PathFinder (JPF), a popular tool for verifying Java programs. We evaluate our technique by generating tests for data structures, refactoring engines, and JPF itself. Our experiments show that test generation using UDITA is faster and leads to test descriptions that are easier to write than in previous frameworks. Moreover, the novel execution mechanism of UDITA is essential for making test generation feasible. Using UDITA, we have discovered a number of previously unknown bugs in Eclipse, NetBeans, Sun javac, and JPF. 1.