Citation Context

...pproach to generate test data [29] or unit tests [30], and dynamic symbolic execution as an extension can overcome a number of problems by combining concrete executions with symbolic execution (e.g., =-=[10]-=-, [21]). This idea has been implemented in tools like DART [10] and CUTE [21], and is also applied in Microsoft’s parametrized unit testing tool PEX [24] or the object-oriented testing framework Symst...

Citation Context

...h process to dive in longer test sequences and then reduce their length during/after the search can provide staggering improvements in term of achieved coverage [2]. • In our GA we use rank selection =-=[28]-=- based on the fitness function (i.e., the obtained coverage and branch distance values). In case of ties, we assign better ranks to smaller test suites. Notice that including the length directly in th...

Citation Context

...parametrized unit testing tool PEX [24] or the object-oriented testing framework Symstra [30]. Meta-heuristic search techniques have been used as an alternative to symbolic execution based approaches =-=[17]-=-. Searchbased techniques have also been applied to test object oriented software using method sequences [?], [25] or strongly typed genetic programming [20], [27]. A promising avenue seems to be the c...

Citation Context

...rete executions with symbolic execution (e.g., [10], [21]). This idea has been implemented in tools like DART [10] and CUTE [21], and is also applied in Microsoft’s parametrized unit testing tool PEX =-=[24]-=- or the object-oriented testing framework Symstra [30]. Meta-heuristic search techniques have been used as an alternative to symbolic execution based approaches [17]. Searchbased techniques have also ...

Citation Context

...lly sized programs fully automatically. A common approach is to select one coverage goal at a time (e.g., a program branch), and to derive a test case that exercises this particular goal (e.g., [14], =-=[25]-=-). Although feasible, there is a major flaw in this strategy, as it assumes that all coverage goals are equally important, equally difficult to reach, and independent of each other. Unfortunately, non...

Citation Context

...erion (e.g., mutation testing [16]) is amenable to automated test generation. Solving path constraints generated with symbolic execution is a popular approach to generate test data [29] or unit tests =-=[30]-=-, and dynamic symbolic execution as an extension can overcome a number of problems by combining concrete executions with symbolic execution (e.g., [10], [21]). This idea has been implemented in tools ...

Citation Context

... test inputs either symbolically or with a search-based approach. The predominant criterion in the literature is branch coverage, but in principle any other coverage criterion (e.g., mutation testing =-=[16]-=-) is amenable to automated test generation. Solving path constraints generated with symbolic execution is a popular approach to generate test data [29] or unit tests [30], and dynamic symbolic executi...

Citation Context

...ototype. For example, EVOSUITE currently has only basic support for arrays and textual inputs, and does not implement all available techniques in searchbased testing (e.g., Testability Transformation =-=[12]-=-, [18]). Implementing these improvements will further improve the results and make EVOSUITE applicable to a wider range of applications. In our empirical study, we targeted object-oriented software. H...

Citation Context

...has filled its internal array. Furthermore, a test case targeting a particular coverage goal will mostly also satisfy further coverage goals by accident 1 public class Stack { 2 int[] values = new int=-=[3]-=-; 3 int size = 0; 4 void push(int x) { 5 if (size >= values.length) ⇐ Requires a full stack 6 resize() ; 7 if (size < values.length) ⇐ Else branch is infeasible 8 values[size++] = x; 9 } 10 int pop() ...

Citation Context

... which does not change during the search. In the literature of testing object oriented software, there are also techniques that are not directly aimed to achieve code coverage, as for example Randoop =-=[19]-=-. In that work, sequences of function calls are generated incrementally using an extension of random testing (for details, see [19]), and the goal is to find test sequences for which the SUT fails. Bu...

Citation Context

...riority of the EVOSUITE technique. Statistical difference has been measured with the MannWhitney U test. To quantify the improvement in a standardized way, we used the Vargha-Delaney Â12 effect size =-=[26]-=-. In our context, the Â12 is an estimation of the probability that, if we run EVOSUITE, we will obtain better coverage than running the single branch strategy. When two randomized algorithms are equi...

Citation Context

...other coverage criterion (e.g., mutation testing [16]) is amenable to automated test generation. Solving path constraints generated with symbolic execution is a popular approach to generate test data =-=[29]-=- or unit tests [30], and dynamic symbolic execution as an extension can overcome a number of problems by combining concrete executions with symbolic execution (e.g., [10], [21]). This idea has been im...

Citation Context

...ftware using method sequences [?], [25] or strongly typed genetic programming [20], [27]. A promising avenue seems to be the combination of evolutionary methods with dynamic symbolic execution (e.g., =-=[15]-=-), alleviating some of the problems both approaches have. Most approaches described in the literature aim to generate test suites that achieve as high as possible branch coverage. In principle, any ot...

Citation Context

...o symbolic execution based approaches [17]. Searchbased techniques have also been applied to test object oriented software using method sequences [?], [25] or strongly typed genetic programming [20], =-=[27]-=-. A promising avenue seems to be the combination of evolutionary methods with dynamic symbolic execution (e.g., [15]), alleviating some of the problems both approaches have. Most approaches described ...

Citation Context

.... All approaches mentioned so far target a single test goal at a time – this is the predominant method. There are some notable exceptions in search based software testing. The works of Arcuri and Yao =-=[6]-=- and Baresi et al. [7] use a single sequence of function calls to maximize the number of covered branches while minimizing the length of such a test case. A drawback of such an approach is that there ...

Citation Context

...istically sized programs fully automatically. A common approach is to select one coverage goal at a time (e.g., a program branch), and to derive a test case that exercises this particular goal (e.g., =-=[14]-=-, [25]). Although feasible, there is a major flaw in this strategy, as it assumes that all coverage goals are equally important, equally difficult to reach, and independent of each other. Unfortunatel...

Citation Context

...ation, we chose five open source libraries. To avoid a bias caused by considering only open source code, we also selected a subset of an industrial case study project previously used by Arcuri et al. =-=[4]-=-. This results in a total of 1,308 classes, which were tested by only calling the API of the 727 public classes (the remaining classes are private and anonymous member classes). The projects were chos...

Citation Context

...oned so far target a single test goal at a time – this is the predominant method. There are some notable exceptions in search based software testing. The works of Arcuri and Yao [6] and Baresi et al. =-=[7]-=- use a single sequence of function calls to maximize the number of covered branches while minimizing the length of such a test case. A drawback of such an approach is that there can be conflicting tes...

Citation Context

...manifests itself in several ways: Many coverage goals are simply infeasible, meaning that there exists no test that would exercise them; this is an instance of the undecidable infeasible path problem =-=[11]-=-. For example, consider the stack implementation in Figure 1: The false branch of the if condition in Line 7 is infeasible. Targeting this goal will per definition fail and the effort was wasted. The ...

Citation Context

...ntation: Some branches are more difficult to cover than others, can lead to coverage of further branches, and some branches can be infeasible. (collateral coverage, also called serendipitous coverage =-=[13]-=-). Again the order in which goals are chosen influences the result – even if all coverage goals are considered, collateral coverage can influence the resulting test suite. For example, covering the tr...

Citation Context

...rage. The search operators for test cases make use of only the type information in the test cluster, and so difficulties can arise when method signatures are imprecise: For example, container classes =-=[22]-=- often declare Object as parameter or return type. Because the search operators have no access to dynamic type information, casting to the appropriate class is difficult. In particular, this problem a...

Citation Context

...nimize: fitness(T ) = |M | − |MT |+ ∑ bk∈B d(bk,T ) (1) D. Bloat Control A variable size representation could lead to bloat, which is a problem that is very typical for example in Genetic Programming =-=[23]-=-: For instance, after each generation, test cases can become longer and longer, until all the memory is consumed, even if shorter sequences are better rewarded. Notice that bloat is an extremely compl...

Citation Context

...e. For example, EVOSUITE currently has only basic support for arrays and textual inputs, and does not implement all available techniques in searchbased testing (e.g., Testability Transformation [12], =-=[18]-=-). Implementing these improvements will further improve the results and make EVOSUITE applicable to a wider range of applications. In our empirical study, we targeted object-oriented software. However...

Citation Context

...h to generate test data [29] or unit tests [30], and dynamic symbolic execution as an extension can overcome a number of problems by combining concrete executions with symbolic execution (e.g., [10], =-=[21]-=-). This idea has been implemented in tools like DART [10] and CUTE [21], and is also applied in Microsoft’s parametrized unit testing tool PEX [24] or the object-oriented testing framework Symstra [30...

Citation Context

...ngle test sequence regardless of its length. Regarding the optimization of an entire test suite in which all test cases are considered at the same time, we are aware of only the work of Baudry et al. =-=[8]-=-. In that work, test suites are optimized with a search algorithm with respect to mutation analysis. However, in that work there is the strong limitation of having to manually choose and fix the lengt...

Citation Context

... – this needs to be searched for. The entire search space of test suites is composed of all possible sets of size from 0 to N (i.e., n ∈ [0,N ]). Each test case can have a size from 1 to L (i.e., l ∈ =-=[1,L]-=-). We need to have these constraints, because in the context addressed in this paper we are not assuming the presence of an automated oracle. Therefore, we cannot expect software testers to manually c...

Citation Context

...tive to symbolic execution based approaches [17]. Searchbased techniques have also been applied to test object oriented software using method sequences [?], [25] or strongly typed genetic programming =-=[20]-=-, [27]. A promising avenue seems to be the combination of evolutionary methods with dynamic symbolic execution (e.g., [15]), alleviating some of the problems both approaches have. Most approaches desc...

Citation Context

...ble values. However, when other metrics are introduced with large domains of possible values (e.g., branch distance and also for example mutation impact [?]), then bloat might occur. In previous work =-=[9]-=-, we have studied several bloat control methods from the literature of Genetic Programming [23] applied in the context of testing object oriented software. However, our previous study [9] covered only...

Citation Context

...er N and L. In fact, allowing the search process to dive in longer test sequences and then reduce their length during/after the search can provide staggering improvements in term of achieved coverage =-=[2]-=-. • In our GA we use rank selection [28] based on the fitness function (i.e., the obtained coverage and branch distance values). In case of ties, we assign better ranks to smaller test suites. Notice ...

Citation Context

...dually, it is important to keep track of the accidental collateral coverage of the remaining targets. Otherwise, it has been proven that random testing would fare better under some scalability models =-=[5]-=-. Recently, Harman et al. [13] proposed a search based multiobjective approach in which, although each goal is still targeted individually, there is the secondary objective of maximizing the number of...