Results 1 - 10
of
38
Feedback-directed random test generation
- In ICSE
, 2007
"... We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soo ..."
Abstract
-
Cited by 177 (17 self)
- Add to MetaCart
(Show Context)
We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soon as an input is built, it is executed and checked against a set of contracts and filters. The result of the execution determines whether the input is redundant, illegal, contract-violating, or useful for generating more inputs. The technique outputs a test suite consisting of unit tests for the classes under test. Passing tests can be used to ensure that code contracts are preserved across program changes; failing tests (that violate one or more contract) point to potential errors that should be corrected. Our experimental results indicate that feedback-directed random test generation can outperform systematic and undirected random test generation, in terms of coverage and error detection. On four small but nontrivial data structures (used previously in the literature), our technique achieves higher or equal block and predicate coverage than model checking (with and without abstraction) and undirected random generation. On 14 large, widely-used libraries (comprising 780KLOC), feedback-directed random test generation finds many previously-unknown errors, not found by either model checking or undirected random generation. 1
Hybrid concolic testing
"... We present hybrid concolic testing, an algorithm that interleaves random testing with concolic execution to obtain both a deep and a wide exploration of program state space. Our algorithm generates test inputs automatically by interleaving random testing until saturation with bounded exhaustive symb ..."
Abstract
-
Cited by 130 (7 self)
- Add to MetaCart
We present hybrid concolic testing, an algorithm that interleaves random testing with concolic execution to obtain both a deep and a wide exploration of program state space. Our algorithm generates test inputs automatically by interleaving random testing until saturation with bounded exhaustive symbolic exploration of program points. It thus combines the ability of random search to reach deep program states quickly together with the ability of concolic testing to explore states in a neighborhood exhaustively. We have implemented our algorithm on top of CUTE and applied it to obtain better branch coverage for an editor implementation (VIM 5.7, 150K lines of code) as well as a data structure implementation in C. Our experiments suggest that hybrid concolic testing can handle large programs and provide, for the same testing budget, almost 4 × the branch coverage than random testing and almost 2 × that of concolic testing.
DSD-Crasher: A hybrid analysis tool for bug finding
- In ISSTA
, 2006
"... DSD-Crasher is a bug finding tool that follows a three-step approach to program analysis: D. Capture the program’s intended execution behavior with dynamic invariant detection. The derived invariants exclude many unwanted values from the program’s input domain. S. Statically analyze the program with ..."
Abstract
-
Cited by 86 (5 self)
- Add to MetaCart
DSD-Crasher is a bug finding tool that follows a three-step approach to program analysis: D. Capture the program’s intended execution behavior with dynamic invariant detection. The derived invariants exclude many unwanted values from the program’s input domain. S. Statically analyze the program within the restricted input domain to explore many paths. D. Automatically generate test cases that focus on reproducing the predictions of the static analysis. Thereby confirmed results are feasible. This three-step approach yields benefits compared to past two-step combinations in the literature. In our evaluation with third-party applications, we demonstrate higher precision over tools that lack a dynamic step and higher efficiency over tools that lack a static step.
Combining unit-level symbolic execution and system-level concrete execution for testing NASA software
- In ISSTA
, 2008
"... peter.c.mehlitz, david.h.bushnell, karen.gundy-burlet, ..."
Abstract
-
Cited by 75 (13 self)
- Add to MetaCart
(Show Context)
peter.c.mehlitz, david.h.bushnell, karen.gundy-burlet,
Atif M Memon. Using GUI Run-Time State as Feedback to Generate Test Cases
- 29th International Conference on Software Engineering (ICSE’07
, 2007
"... This paper presents a new automated model-driven tech-nique to generate test cases by using feedback from the ex-ecution of a seed test suite on an application under test (AUT). The test cases in the seed suite are designed to be generated automatically and executed very quickly. During their execu ..."
Abstract
-
Cited by 50 (13 self)
- Add to MetaCart
(Show Context)
This paper presents a new automated model-driven tech-nique to generate test cases by using feedback from the ex-ecution of a seed test suite on an application under test (AUT). The test cases in the seed suite are designed to be generated automatically and executed very quickly. During their execution, feedback obtained from the AUT's run-time state is used to generate new, improved test cases. The new test cases subsequently become part of the seed suite. This anytime technique continues iteratively, generating and executing additional test cases until resources are ex-hausted or testing goals have been met. The feedback-based technique is demonstrated for auto-mated testing of graphical user interfaces (GUIs). An ex-isting abstract model of the GUI is used to automatically generate the seed test suite. It is executed; during its ex-ecution, state changes in the GUI pinpoint important rela-tionships between GUI events, which evolve the model and help to generate new test cases. Together with a reverse-engineering algorithm used to obtain the initial model and seed suite, the feedback-based technique yields a fully au-tomatic, end-to-end GUI testing process. A feasibility study on four large elded open-source software (OSS) applica-tions demonstrates that this process is able to signicantly improve existing techniques and help identify/report serious problems in the OSS. In response, these problems have been xed by the developers of the OSS in subsequent versions. 1
Search based software testing of object-oriented containers
- Information Sciences
, 2008
"... Automatic software testing tools are still far from ideal for real world Object-Oriented (OO) Software. The use of nature inspired search algorithms for this problem has been investigated recently. Testing complex data structures (e.g., containers) is very challenging since testing software with sim ..."
Abstract
-
Cited by 32 (12 self)
- Add to MetaCart
(Show Context)
Automatic software testing tools are still far from ideal for real world Object-Oriented (OO) Software. The use of nature inspired search algorithms for this problem has been investigated recently. Testing complex data structures (e.g., containers) is very challenging since testing software with simple states is already hard. Because containers are used in almost every type of software, their reliability is of utmost importance. Hence, this paper focuses on the difficulties of testing container classes with nature inspired search algorithms. We will first describe how input data can be automatically generated for testing Java Containers. Input space reductions and a novel testability transformation are presented to aid the search algorithms. Different search algorithms are then considered and studied in order to understand when and why a search algorithm is effective for a testing problem. In our experiments, these nature inspired search algorithms seem to give better results than the traditional techniques described in literature. Besides, the problem of minimising the length of the test sequences is also addressed. Finally, some open research questions are given.
A.: Sound empirical evidence in software testing
- In: ACM/IEEE International Conference on Software Engineering (ICSE
, 2012
"... Abstract—Several promising techniques have been proposed to automate different tasks in software testing, such as test data generation for object-oriented software. However, reported studies in the literature only show the feasibility of the proposed techniques, because the choice of the employed ar ..."
Abstract
-
Cited by 25 (17 self)
- Add to MetaCart
(Show Context)
Abstract—Several promising techniques have been proposed to automate different tasks in software testing, such as test data generation for object-oriented software. However, reported studies in the literature only show the feasibility of the proposed techniques, because the choice of the employed artifacts in the case studies (e.g., software applications) is usually done in a non-systematic way. The chosen case study might be biased, and so it might not be a valid representative of the addressed type of software (e.g., internet applications and embedded systems). The common trend seems to be to accept this fact and get over it by simply discussing it in a threats to validity section. In this paper, we evaluate search-based software testing (in particular the EVOSUITE tool) when applied to test data generation for open source projects. To achieve sound empirical results, we randomly selected 100 Java projects from SourceForge, which is the most popular open source repository (more than 300,000 projects with more than two million registered users). The resulting case study not only is very large (8784 public classes for a total of 291,639 bytecode level branches), but more importantly it is statistically sound and representative for open source projects. Results show that while high coverage on commonly used types of classes is achievable, in practice environmental dependencies prohibit such high coverage, which clearly points out essential future research directions. To support this future research, our SF100 case study can serve as a much needed benchmark for test generation. Keywords-test case generation; unit testing; search-based software engineering; benchmark I.
Testing container classes: Random or systematic?
- IN FUNDAMENTAL APPROACHES TO SOFTWARE ENGINEERING
"... Container classes such as lists, sets, or maps are elementary data structures common to many programming languages. Since they are a part of standard libraries, they are important to test, which led to research on advanced testing techniques targeting such containers and research on comparing test ..."
Abstract
-
Cited by 21 (11 self)
- Add to MetaCart
(Show Context)
Container classes such as lists, sets, or maps are elementary data structures common to many programming languages. Since they are a part of standard libraries, they are important to test, which led to research on advanced testing techniques targeting such containers and research on comparing testing techniques using such containers. However, these techniques have not been thoroughly compared to simpler techniques such as random testing. We present the results of a larger case study in which we compare random testing with shape abstraction, a systematic technique that showed the best results in a previous study. Our experiments show that random testing is about as effective as shape abstraction for testing these containers, which raises the question whether containers are well suited as a benchmark for comparing advanced testing techniques.
On the Predictability of Random Tests for Object-Oriented Software
- INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION, AND VALIDATION
, 2008
"... Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detect ..."
Abstract
-
Cited by 19 (9 self)
- Add to MetaCart
(Show Context)
Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detected by random testing over time. We present the results of an empirical study that is based on 1215 hours of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. Analyzing over 6 million failures triggered during the experiments, the study provides evidence that the relative number of faults detected by random testing over time is predictable but that different runs of the random test case generator detect different faults. The study also shows that random testing quickly finds faults: the first failure is likely to be triggered within 30 seconds.
Generating event sequence-based test cases using GUI runtime state feedback
- IEEE Transactions on Software Engineering
"... Abstract—This paper presents a fully automatic model-driven technique to generate test cases for Graphical user interface-based applications (GUIs). The technique uses feedback from the execution of a “seed test suite, ” which is generated automatically using an existing structural event-interaction ..."
Abstract
-
Cited by 17 (1 self)
- Add to MetaCart
(Show Context)
Abstract—This paper presents a fully automatic model-driven technique to generate test cases for Graphical user interface-based applications (GUIs). The technique uses feedback from the execution of a “seed test suite, ” which is generated automatically using an existing structural event-interaction graph model of the GUI. During its execution, the run-time effect of each GUI event on all other events pinpoints event-semantic interaction (ESI) relationships, which are used to automatically generate new test cases. Two studies on eight applications demonstrate that the feedback-based technique (1) is able to significantly improve existing techniques and help identify serious problems in the software and (2) the ESI relationships captured via GUI state yield test suites that most often detect more faults than their code-, event-, and event-interaction-coverage equivalent counterparts. Index Terms—GUI testing, automated testing, model-based testing, GUITAR testing system. F
