Results 1 - 10
of
25
A.: Evosuite: Automatic test suite generation for object-oriented software
- In: Proc. of ACM SIGSOFT ESEC/FSE
, 2011
"... To find defects in software, one needs test cases that ex-ecute the software systematically, and oracles that assess the correctness of the observed behavior when running these test cases. This paper presents EvoSuite, a tool that au-tomatically generates test cases with assertions for classes writt ..."
Abstract
-
Cited by 62 (19 self)
- Add to MetaCart
(Show Context)
To find defects in software, one needs test cases that ex-ecute the software systematically, and oracles that assess the correctness of the observed behavior when running these test cases. This paper presents EvoSuite, a tool that au-tomatically generates test cases with assertions for classes written in Java code. To achieve this, EvoSuite applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion. For the produced test suites, EvoSuite suggests possible oracles by adding small and effective sets of assertions that concisely summarize the current behavior; these assertions allow the developer to detect deviations from expected behavior, and to capture the current behavior in order to protect against future defects breaking this behavior.
A.: Sound empirical evidence in software testing
- In: ACM/IEEE International Conference on Software Engineering (ICSE
, 2012
"... Abstract—Several promising techniques have been proposed to automate different tasks in software testing, such as test data generation for object-oriented software. However, reported studies in the literature only show the feasibility of the proposed techniques, because the choice of the employed ar ..."
Abstract
-
Cited by 25 (17 self)
- Add to MetaCart
(Show Context)
Abstract—Several promising techniques have been proposed to automate different tasks in software testing, such as test data generation for object-oriented software. However, reported studies in the literature only show the feasibility of the proposed techniques, because the choice of the employed artifacts in the case studies (e.g., software applications) is usually done in a non-systematic way. The chosen case study might be biased, and so it might not be a valid representative of the addressed type of software (e.g., internet applications and embedded systems). The common trend seems to be to accept this fact and get over it by simply discussing it in a threats to validity section. In this paper, we evaluate search-based software testing (in particular the EVOSUITE tool) when applied to test data generation for open source projects. To achieve sound empirical results, we randomly selected 100 Java projects from SourceForge, which is the most popular open source repository (more than 300,000 projects with more than two million registered users). The resulting case study not only is very large (8784 public classes for a total of 291,639 bytecode level branches), but more importantly it is statistically sound and representative for open source projects. Results show that while high coverage on commonly used types of classes is achievable, in practice environmental dependencies prohibit such high coverage, which clearly points out essential future research directions. To support this future research, our SF100 case study can serve as a much needed benchmark for test generation. Keywords-test case generation; unit testing; search-based software engineering; benchmark I.
Search-based system testing: High coverage, no false alarms
- In International Symposium on Software Testing and Analysis (ISSTA
, 2012
"... Modern test case generation techniques can automatically achieve high code coverage. If they operate on the unit level, they run the risk of generating inputs infeasible in reality, which, when causing failures, are painful to identify and eliminate. Running a unit test generator on five open source ..."
Abstract
-
Cited by 25 (3 self)
- Add to MetaCart
(Show Context)
Modern test case generation techniques can automatically achieve high code coverage. If they operate on the unit level, they run the risk of generating inputs infeasible in reality, which, when causing failures, are painful to identify and eliminate. Running a unit test generator on five open source Java programs, we found that all of the 181 reported failures were false failures—that is, indicating a problem in the generated test case rather than the program. By generating test cases at the GUI level, our EXSYST prototype can avoid such false alarms by construction. In our evaluation, it achieves higher coverage than search-based test generators at the unit level; yet, every failure can be shown to be caused by a real sequence of input events. Whenever a system interface is available, we recommend considering search-based system testing as an alternative to avoid false failures.
On parameter tuning in search based software engineering
- In Proc. SSBSE
, 2011
"... Abstract. When applying search-based software engineering (SBSE) techniques one is confronted with a multitude of different parameters that need to be chosen: Which population size for a genetic algorithm? Which selection mechanism to use? What settings to use for dozens of other parameters? This pr ..."
Abstract
-
Cited by 21 (10 self)
- Add to MetaCart
(Show Context)
Abstract. When applying search-based software engineering (SBSE) techniques one is confronted with a multitude of different parameters that need to be chosen: Which population size for a genetic algorithm? Which selection mechanism to use? What settings to use for dozens of other parameters? This problem not only troubles users who want to apply SBSE tools in practice, but also researchers performing experimentation – how to compare algorithms that can have different parameter settings? To shed light on the problem of parameters, we performed the largest empirical analysis on parameter tuning in SBSE to date, collecting and statistically analysing data from more than a million experiments. As case study, we chose test data generation, one of the most popular problems in SBSE. Our data confirm that tuning does have a critical impact on algorithmic perfor-mance, and over-fitting of parameter tuning is a dire threat to external validity of empirical analyses in SBSE. Based on this large empirical evidence, we give guidelines on how to handle parameter tuning.
Combining search-based and constraint-based testing
- In IEEE/ACM Int. Conference on Automated Software Engineering (ASE
, 2011
"... Abstract—Many modern automated test generators are based on either meta-heuristic search techniques or use constraint solvers. Both approaches have their advantages, but they also have specific drawbacks: Search-based methods get stuck in local optima and degrade when the search landscape offers no ..."
Abstract
-
Cited by 15 (7 self)
- Add to MetaCart
(Show Context)
Abstract—Many modern automated test generators are based on either meta-heuristic search techniques or use constraint solvers. Both approaches have their advantages, but they also have specific drawbacks: Search-based methods get stuck in local optima and degrade when the search landscape offers no guidance; constraint-based approaches, on the other hand, can only handle certain domains efficiently. In this paper we describe a method that integrates both techniques and delivers the best of both worlds. On a high-level view, our method uses a genetic algorithm to generate tests, but the twist is that during evolution a constraint solver is used to ensure that mutated offspring efficiently explores different control flow. Experiments on 20 case study examples show that on average the combination improves branch coverage by 28 % over search-based techniques and by 13 % over constraint-based techniques. I.
A.: EXSYST: Search-based GUI testing
- In: ICSE
, 2012
"... Abstract—Test generation tools commonly aim to cover structural artefacts of software, such as either the source code or the user interface. However, focusing only on source code can lead to unrealistic or irrelevant test cases, while only exploring a user interface often misses much of the underlyi ..."
Abstract
-
Cited by 9 (2 self)
- Add to MetaCart
(Show Context)
Abstract—Test generation tools commonly aim to cover structural artefacts of software, such as either the source code or the user interface. However, focusing only on source code can lead to unrealistic or irrelevant test cases, while only exploring a user interface often misses much of the underlying program behavior. Our EXSYST prototype takes a new approach by exploring user interfaces while aiming to maximize code coverage, thus combining the best of both worlds. Experiments show that such an approach can achieve high code coverage matching and exceeding the code coverage of traditional unit-based test generators; yet, by construction every test case is realistic and relevant, and every detected failure can be shown to be caused by a real sequence of input events. Keywords-test case generation; system testing; GUI testing; test coverage
Isolating Failure Causes through Test Case Generation
"... Manual debugging is driven by experiments—test runs that nar-row down failure causes by systematically confirming or excluding individual factors. The BUGEX approach leverages test case gen-eration to systematically isolate such causes from a single failing test run—causes such as properties of exec ..."
Abstract
-
Cited by 8 (0 self)
- Add to MetaCart
(Show Context)
Manual debugging is driven by experiments—test runs that nar-row down failure causes by systematically confirming or excluding individual factors. The BUGEX approach leverages test case gen-eration to systematically isolate such causes from a single failing test run—causes such as properties of execution states or branches taken that correlate with the failure. Identifying these causes al-lows for deriving conclusions as: “The failure occurs whenever the daylight savings time starts at midnight local time. ” In our evalua-tion, a prototype of BUGEX precisely pinpointed important failure explaining facts for six out of seven real-life bugs.
A Parallel Genetic Algorithm Based on Hadoop MapReduce for the Automatic Generation of JUnit Test Suites
"... Abstract — Software testing represents one of the most explored fields of application of Search-Based techniques and a range of testing problems have been successfully addressed using Genetic Algorithms. Nevertheless, to date Search-Based Software Testing (SBST) has found limited application in indu ..."
Abstract
-
Cited by 7 (2 self)
- Add to MetaCart
(Show Context)
Abstract — Software testing represents one of the most explored fields of application of Search-Based techniques and a range of testing problems have been successfully addressed using Genetic Algorithms. Nevertheless, to date Search-Based Software Testing (SBST) has found limited application in industry. As in other fields of Search-Based Software Engineering, this is principally due to the fact that when applied to large problems, Search-Based approaches may require too much computational efforts. In this scenario, parallelization may be a suitable way to improve the performance especially due to the fact that many of these techniques are “naturally parallelizable”. Nevertheless, very few attempts have been provided for SBST parallelization. In this paper, we present a Parallel Genetic Algorithm for the automatic generation of test suites. The solution is based on Hadoop MapReduce since it is well supported to work also in the cloud and on graphic cards, thus being an ideal candidate for high scalable parallelization of Genetic Algorithms. A preliminary analysis of the proposal was carried out aiming to evaluate the speed-up with respect to the sequential execution. The analysis was based on a real world open source library.
The role of artificial intelligence in software engineering
- In First International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering
, 2012
"... Abstract—There has been a recent surge in interest in the ..."
Abstract
-
Cited by 5 (1 self)
- Add to MetaCart
(Show Context)
Abstract—There has been a recent surge in interest in the
A.: EvoSuite: On the challenges of test case generation in the real world (tool paper
- In: IEEE International Conference on Software Testing, Verification and Validation (ICST
, 2013
"... Abstract—Test case generation is an important but tedious task, such that researchers have devised many different proto-types that aim to automate it. As these are research prototypes, they are usually only evaluated on a few hand-selected case studies, such that despite great results there remains ..."
Abstract
-
Cited by 3 (3 self)
- Add to MetaCart
(Show Context)
Abstract—Test case generation is an important but tedious task, such that researchers have devised many different proto-types that aim to automate it. As these are research prototypes, they are usually only evaluated on a few hand-selected case studies, such that despite great results there remains the question of usability in the “real world”. EVOSUITE is such a research prototype, which automatically generates unit test suites for classes written in the Java programming language. In our ongoing endeavour to achieve real-world usability, we recently passed the milestone success of applying EVOSUITE on hundred projects randomly selected from the SourceForge open source platform. This paper discusses the technical challenges that a testing tool like EVOSUITE needs to address when handling Java classes coming from real-world open source projects, and when producing JUnit test suites intended for real users. Keywords-test case generation; search-based testing; testing classes; search-based software engineering I.
