Results 1 - 10
of
20
An improved meta-heuristic search for constrained interaction testing
- in Proceedings of International Symposiumon Search-Based Software Engineering
, 2009
"... Combinatorial interaction testing (CIT) is a costeffective sampling technique for discovering interaction faults in highly configurable systems. Recent work with greedy CIT algorithms efficiently supports constraints on the features that can coexist in a configuration. But when testing a single syst ..."
Abstract
-
Cited by 13 (3 self)
- Add to MetaCart
(Show Context)
Combinatorial interaction testing (CIT) is a costeffective sampling technique for discovering interaction faults in highly configurable systems. Recent work with greedy CIT algorithms efficiently supports constraints on the features that can coexist in a configuration. But when testing a single system configuration is expensive, greedy techniques perform worse than meta-heuristic algorithms because they produce larger samples. Unfortunately, current meta-heuristic algorithms are inefficient when constraints are present. We investigate the sources of inefficiency, focusing on simulated annealing, a well-studied meta-heuristic algorithm. From our findings we propose changes to improve performance, including a reorganized search space based on the CIT problem structure. Our empirical evaluation demonstrates that the optimizations reduce run-time by three orders of magnitude and yield smaller samples. Moreover, on real problems the new version compares favorably with greedy algorithms. 1.
A formal logic approach to constrained combinatorial testing
- JOURNAL OF AUTOMATED REASONING
"... ..."
Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize t-wise Test Suites for Large Software Product Lines
, 2012
"... Software Product Lines (SPLs) are families of prod-ucts whose commonalities and variability can be captured by Feature Models (FMs). T-wise testing aims at finding errors triggered by all interactions amongst t features, thus reduc-ing drastically the number of products to test. T-wise testing appro ..."
Abstract
-
Cited by 9 (0 self)
- Add to MetaCart
(Show Context)
Software Product Lines (SPLs) are families of prod-ucts whose commonalities and variability can be captured by Feature Models (FMs). T-wise testing aims at finding errors triggered by all interactions amongst t features, thus reduc-ing drastically the number of products to test. T-wise testing approaches for SPLs are limited to small values of t – which miss faulty interactions – or limited by the size of the FM. Furthermore, they neither prioritize the products to test nor provide means to finely control the generation process. This paper offers (a) a search-based approach capable of generating products for large SPLs, forming a scalable and flexible alternative to current techniques and (b) prioritization algorithms for any set of products. Experiments conducted on 124 FMs (including large FMs such as the Linux kernel) demonstrate the feasibility and the practicality of our approach.
Using binary decision diagrams for combinatorial test design
- In: Proc. of the 2011 International Symposium on Software Testing and Analysis (ISSTA’11
, 2011
"... ABSTRACT Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interaction in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then autom ..."
Abstract
-
Cited by 8 (0 self)
- Add to MetaCart
(Show Context)
ABSTRACT Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interaction in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. Various combinatorial testing tools exist, implementing different approaches to finding a set of tests that satisfies t-wise coverage. However, little consideration has been given to the process of defining the test space for CTD, which is usually a manual, labor-intensive, and error-prone effort. Potential errors include missing parameters and their values, wrong identification of parameters and of valid value combinations, and errors in the definition of restrictions that cause them not to capture the intended combinations. From our experience, lack of support for the test space definition process is one of the main obstacles in applying CTD to a wide range of testing domains. In this work, we present a Cartesian product based methodology and technology that assist in defining a complete and consistent test space for CTD. We then show how using binary decision diagrams (BDDs) to represent and build the test space dramatically increases the scalability of our approach, making it applicable to large and complex real-life test design tasks, for which explicit representation of the test space is infeasible. Finally, we show how BDDs can be used also to solve the CTD problem itself. We present a new and highly effective BDD-based approach for solving CTD, which finds a set of tests that satisfies t-wise coverage by subset selection. Our approach supports also advanced requirements such as requirements on the distribution of values in the selected tests. We apply our algorithm to real-life testing problems of varying complexity, and show its superior performance.
Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing
"... Combinatorial interaction testing aims at revealing errors inside a system under test triggered by unintended interaction between values of its input parameters. In this context we defined a new greedy approach to generate a combinatorial interaction test suites in the presence of constraints, based ..."
Abstract
-
Cited by 8 (4 self)
- Add to MetaCart
Combinatorial interaction testing aims at revealing errors inside a system under test triggered by unintended interaction between values of its input parameters. In this context we defined a new greedy approach to generate a combinatorial interaction test suites in the presence of constraints, based on integration of an SMT solver, and ordered processing of test goals. Based on the observation that the processing order of required combinations determines the size of the final test suite, this approach has been then used as a framework to evaluate a set of deterministic ordering strategies, each based on a different heuristic optimization criteria. Their performance has been assessed and contrasted also with those of random and dummy ordering strategies. Results of experimental assessment are presented and compared with well-known combinatorial tools.
Interaction coverage meets path coverage by smt constraint solving
- of Lecture Notes in Computer Science
, 2009
"... Abstract. We present a novel approach for generating interaction combinations based on SMT constraint resolution. Our approach can generate maximal inter-action coverage in the presence of general constraints as supported by the un-derlying solver. It supports seeding with general predicates, which ..."
Abstract
-
Cited by 7 (0 self)
- Add to MetaCart
(Show Context)
Abstract. We present a novel approach for generating interaction combinations based on SMT constraint resolution. Our approach can generate maximal inter-action coverage in the presence of general constraints as supported by the un-derlying solver. It supports seeding with general predicates, which allows us to combine it with path exploration such that both interaction and path coverage goals can be met. Our approach is motivated by the application to behavioral model-based testing in the Spec Explorer tool, where parameter combinations must be generated such that all path conditions of a model action have at least one combination which enables the path. It is applied in a large-scale project for model-based quality assurance of interoperability documentation at Microsoft. 1
Experiments on the Test Case Length in Specification Based Test Case Generation
"... Many different techniques have been proposed to address the problem of automated test case generation, varying in a range of properties and resulting in very different test cases. In this paper we investigate the effects of the test case length on resulting test suites: Intuitively, longer test case ..."
Abstract
-
Cited by 5 (2 self)
- Add to MetaCart
(Show Context)
Many different techniques have been proposed to address the problem of automated test case generation, varying in a range of properties and resulting in very different test cases. In this paper we investigate the effects of the test case length on resulting test suites: Intuitively, longer test cases should serve to find more difficult faults but will reduce the number of test cases necessary to achieve the test objectives. On the other hand longer test cases have disadvantages such as higher computational costs and they are more difficult to interpret manually. Consequently, should one aim to generate many short test cases or fewer but longer test cases? We present the results of a set of experiments performed in a scenario of specification based testing for reactive systems. As expected, a long test case can achieve higher coverage and fault detecting capability than a short one, while giving preference to longer test cases in general can help reduce the size of test suites but can also have the opposite effect, for example, if minimization is applied. 1.
An Evaluation of Model Checkers for Specification Based Test Case Generation
"... Under certain constraints the test case generation problem can be represented as a model checking problem, thus enabling the use of powerful model checking tools to perform the test case generation automatically. There are, however, several different model checking techniques, and to date there is l ..."
Abstract
-
Cited by 5 (3 self)
- Add to MetaCart
(Show Context)
Under certain constraints the test case generation problem can be represented as a model checking problem, thus enabling the use of powerful model checking tools to perform the test case generation automatically. There are, however, several different model checking techniques, and to date there is little evidence and comparison on which of these techniques is best suited for test case generation. This paper presents the results of an evaluation of several different model checkers on a set of realistic formal specifications given in the SCR [21] notation. For each specification test cases are generated for a set of coverage criteria with each of the model checkers using different configurations. The evaluation shows that the best suited model checking technique and optimization very much depend on the specification that is used to generate test cases. However, from the experiments we can draw general conclusions about which optimizations are useful and which model checking technique is best suited for which type of model. Finally, we demonstrate that by combining several model checking techniques it is possible to significantly speed up test case generation and also achieve full test coverage for cases where none of the techniques by itself would succeed. 1.
Software Input Space Modeling with Constraints among Parameters
"... Abstract—This paper considers the task of software test case generation from a large space of values of input parameters. The purpose of the paper is to create a model of software input space with constraints among parameters to serve as a basis of testing. We suggest a procedure to create a directe ..."
Abstract
-
Cited by 3 (2 self)
- Add to MetaCart
(Show Context)
Abstract—This paper considers the task of software test case generation from a large space of values of input parameters. The purpose of the paper is to create a model of software input space with constraints among parameters to serve as a basis of testing. We suggest a procedure to create a directed graph model, where paths through the graph represent all valid (and only valid) input combinations. The procedure accommodates an arbitrary set of dependencies among parameters. It starts from a simple linear graph and sequentially modifies this graph for each dependency between parameters. Modifications include subgraph splitting and elimination of dead nodes and edges. A complete example of a system with six input parameters and five dependencies among them is presented to illustrate the application of the procedure. Applicability of the approach for different types of parameters and dependencies is addressed.
Model-based testing of industrial transformational systems
- In Testing Software and Systems - 23rd IFIP WG 6.1 International Conference, ICTSS 2011
"... Abstract. We present an approach for modeling and testing transformational systems in an industrial context. The systems are modeled as a set of boolean formulas. Each formula is called a clause and is an expression for an expected output value. To manage complexities of the models, we employ a mod ..."
Abstract
-
Cited by 2 (0 self)
- Add to MetaCart
(Show Context)
Abstract. We present an approach for modeling and testing transformational systems in an industrial context. The systems are modeled as a set of boolean formulas. Each formula is called a clause and is an expression for an expected output value. To manage complexities of the models, we employ a modeling trick for handling dependencies, by using some output values from the system under test to verify other output values. To avoid circular dependencies, the clauses are arranged in a hierarchy, where each clause depends on the outputs of its children. This modeling trick enables us to model and test complex systems, using relatively simple models. Pairwise testing is used for test case generation. This manages the number of test cases for complex systems. The approach is developed based on a case study for testing printer controllers in professional printers at Océ. The model-based testing approach results in increased maintainability and gives better understanding of test cases and their produced output. Using pairwise testing resulted in measurable coverage, with a test set smaller than the manually created test set. To illustrate the applicability of the approach, we show how the approach can be used to model and test parts of a controller for ventilation in livestock stables.
