Results 11 - 20
of
43
Combinatorial Interaction Testing with CITLAB
"... Abstract—In this paper the CITLAB tool for Combinatorial Interaction Testing is presented. The tool allows importing/exporting models of combinatorial problems from/to different application domains, by means of a common interchange syntax notation and a corresponding interoperable semantic metamodel ..."
Abstract
-
Cited by 5 (4 self)
- Add to MetaCart
(Show Context)
Abstract—In this paper the CITLAB tool for Combinatorial Interaction Testing is presented. The tool allows importing/exporting models of combinatorial problems from/to different application domains, by means of a common interchange syntax notation and a corresponding interoperable semantic metamodel. Moreover, the tool is a framework allowing embedding and transparent invocation of multiple, different implementations of combinatorial algorithms. CITLAB has been designed tightly integrated with the Eclipse IDE framework, by means of its plug-in extension mechanism. It is intended to easy the spread of CIT testing both in industrial practice and in academic research, by allowing users and researchers to apply multiple test suite generation algorithms, each with its peculiarities, on the same problem models, and let them compare the results in order to select the one that best fits their needs, while alleviating from the pain of knowing all the different details and notations of the underlying CIT tools. Index Terms—Combinatorial testing model, domain-specific language, Eclipse, XTEXT. I.
Reducing Field Failures in System Configurable Software: Cost-Based
"... System testing of configurable software is an expensive and resource constrained process. Insufficient testing often leads to escaped faults in the field where failures impact customers and are costly to repair. Prior work has shown that it is possible to efficiently sample configurations for testin ..."
Abstract
-
Cited by 5 (1 self)
- Add to MetaCart
(Show Context)
System testing of configurable software is an expensive and resource constrained process. Insufficient testing often leads to escaped faults in the field where failures impact customers and are costly to repair. Prior work has shown that it is possible to efficiently sample configurations for testing using combinatorial interaction testing, and to prioritize these configurations to increase the rate of early fault detection. The underlying assumption to date has been that there is no added complexity to configuring a system level environment over a user configurable one; i.e. the time required to setup and test each individual configuration is nominal. In this paper we examine prioritization of system configurable software driven not only by fault detection but also by the cost of configuration and setup time that moving between different configurations incurs. We present a case study on two releases of an enterprise software system using failures reported in the field. We examine the most effective prioritization technique and conclude that (1) using failure history of configurations can improve the early fault detection rate, but that (2) we must consider fault detection rate over time, not by the number of configurations tested. It is better to test related configurations which incur minimal setup time than to test fewer, more diverse configurations. 1.
iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees
"... Abstract—Software configurability has many benefits, but it also makes programs much harder to test, as in the worst case the program must be tested under every possible configuration. One potential remedy to this problem is combinatorial interaction testing (CIT), in which typically the developer s ..."
Abstract
-
Cited by 5 (1 self)
- Add to MetaCart
(Show Context)
Abstract—Software configurability has many benefits, but it also makes programs much harder to test, as in the worst case the program must be tested under every possible configuration. One potential remedy to this problem is combinatorial interaction testing (CIT), in which typically the developer selects a strength t and then computes a covering array containing all t-way configuration option combinations. However, in a prior study we showed that several programs have important highstrength interactions (combinations of a subset of configuration options) that CIT is highly unlikely to generate in practice. In this paper, we propose a new algorithm called interaction tree discovery (iTree) that aims to identify sets of configurations to test that are smaller than those generated by CIT, while also including important high-strength interactions missed by practical applications of CIT. On each iteration of iTree, we first use low-strength CIT to test the program under a set of configurations, and then apply machine learning techniques to discover new interactions that are potentially responsible for any new coverage seen. By repeating this process, iTree builds up a set of configurations likely to contain key high-strength interactions. We evaluated iTree by comparing the coverage it achieves versus covering arrays and randomly generated configuration sets. Our results strongly suggest that iTree can identify high-coverage sets of configurations more effectively than traditional CIT or random sampling.
CITLAB: a Laboratory for Combinatorial Interaction Testing
"... Abstract—Although the research community around combinatorial interaction testing has been very active for several years, it has failed to find common solutions on some issues. First of all, there is not a common abstract nor concrete language to express combinatorial problems. Combinatorial testing ..."
Abstract
-
Cited by 4 (4 self)
- Add to MetaCart
(Show Context)
Abstract—Although the research community around combinatorial interaction testing has been very active for several years, it has failed to find common solutions on some issues. First of all, there is not a common abstract nor concrete language to express combinatorial problems. Combinatorial testing generator tools are strongly decoupled making difficult their interoperability and the exchange of models and data. In this paper, we propose an abstract and concrete specific language for combinatorial problems. It features and formally defines the concepts of parameters and types, constraints, seeds, and test goals. The language is defined by means of XTEXT, a framework for the definition of domain-specific languages. XTEXT is used to derive a powerful editor integrated with eclipse and with all the expected features of a modern editor. Eclipse is also used to build an extensible framework in which test generators, importers, and exporters can be easily added as plugins. Index Terms—Combinatorial testing model, domain-specific language, eclipse, XTEXT. I.
Evolutionary algorithm for prioritized pairwise test data generation
, 2012
"... ABSTRACT Combinatorial Interaction Testing (CIT) is a technique used to discover faults caused by parameter interactions in highly configurable systems. These systems tend to be large and exhaustive testing is generally impractical. Indeed, when the resources are limited, prioritization of test cas ..."
Abstract
-
Cited by 4 (1 self)
- Add to MetaCart
(Show Context)
ABSTRACT Combinatorial Interaction Testing (CIT) is a technique used to discover faults caused by parameter interactions in highly configurable systems. These systems tend to be large and exhaustive testing is generally impractical. Indeed, when the resources are limited, prioritization of test cases is a must. Important test cases are assigned a high priority and should be executed earlier. On the one hand, the prioritization of test cases may reveal faults in early stages of the testing phase. But, on the other hand the generation of minimal test suites that fulfill the demanded coverage criteria is an NP-hard problem. Therefore, search based approaches are required to find the (near) optimal test suites. In this work we present a novel evolutionary algorithm to deal with this problem. The experimental analysis compares five techniques on a set of benchmarks. It reveals that the evolutionary approach is clearly the best in our comparison. The presented algorithm can be integrated into CTE XL professional tool.
Answer-Set Programming as a new Approach to Event-Sequence Testing
"... Abstract—In many applications, faults are triggered by events that occur in a particular order. Based on the assumption that most bugs are caused by the interaction of a low number of events, Kuhn et al. recently introduced sequence covering arrays (SCAs) as suitable designs for event sequence testi ..."
Abstract
-
Cited by 3 (2 self)
- Add to MetaCart
(Show Context)
Abstract—In many applications, faults are triggered by events that occur in a particular order. Based on the assumption that most bugs are caused by the interaction of a low number of events, Kuhn et al. recently introduced sequence covering arrays (SCAs) as suitable designs for event sequence testing. In practice, directly applying SCAs for testing is often impaired by additional constraints, and SCAs have to be adapted to fit application-specific needs. Modifying precomputed SCAs to account for problem variations can be problematic, if not impossible, and developing dedicated algorithms is costly. In this paper, we propose answer-set programming (ASP), a wellknown knowledge-representation formalism from the area of artificial intelligence based on logic programming, as a declarative paradigm for computing SCAs. Our approach allows to concisely state complex coverage criteria in an elaboration tolerant way, i.e., small variations of a problem specification require only small modifications of the ASP representation. Keywords-event-sequence testing; combinatorial interaction testing; answer-set programming. I.
Design of prioritized N-wise testing
- In Proc. of ICTSS’14, LNCS 8763
, 2014
"... Abstract. N-wise testing is a widely used technique for combinato-rial interaction testing. Prioritizing testing reorders test cases by rele-vance, testing important aspects more thoroughly. We propose a novel technique for N-wise test case generation to satisfy the three distinct prioritization cri ..."
Abstract
-
Cited by 2 (1 self)
- Add to MetaCart
(Show Context)
Abstract. N-wise testing is a widely used technique for combinato-rial interaction testing. Prioritizing testing reorders test cases by rele-vance, testing important aspects more thoroughly. We propose a novel technique for N-wise test case generation to satisfy the three distinct prioritization criteria of interaction coverage, weight coverage, and KL divergence. The proposed technique generates small N-wise test cases, where high-priority test cases appear early and frequently. Our early evaluation confirms that the proposed technique improves on existing techniques based on the three prioritization criteria.
Finding interaction faults adaptively using distance-based strategies
- in Proceedings of the 18th IEEE International Conference and Workshops on Engineering of Computer-Based Systems (ECBS ’11
, 2011
"... Abstract-Software systems are typically large and exhaustive testing of all possible input parameters is usually not feasible. Testers select tests that they anticipate may catch faults, yet many unanticipated faults may be overlooked. This work complements current testing methodologies by adaptive ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Abstract-Software systems are typically large and exhaustive testing of all possible input parameters is usually not feasible. Testers select tests that they anticipate may catch faults, yet many unanticipated faults may be overlooked. This work complements current testing methodologies by adaptively dispensing one-test-at-a-time, where each test is as "distant" as possible from previous tests. Two types of distance measures are explored: (1) distance defined in relation to combinations of parameter-values not previously tested together and (2) distance computed as the maximum minimal Hamming distance from previous tests. Experiments compare the effectiveness of these two types of distance-based tests and random tests. Experiments include simulations, as well as examination of instrumented data from an actual system, the Traffic Collision Avoidance System (TCAS). Results demonstrate that the two instantiations of distance-based tests often find more faults sooner and in fewer tests than randomly generated tests.
A Study in Prioritization for Higher Strength Combinatorial Testing
"... Abstract—Recent studies have shown that combinatorial interaction testing (CIT) is an effective fault detection technique and that early fault detection can be improved by ordering test suites by interaction based prioritization approaches. Despite research that has shown that higher strength CIT im ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Abstract—Recent studies have shown that combinatorial interaction testing (CIT) is an effective fault detection technique and that early fault detection can be improved by ordering test suites by interaction based prioritization approaches. Despite research that has shown that higher strength CIT improves fault detection, there have been fewer studies that aim to understand the impact of prioritization based on higher strength criteria. In this paper, we aim to understand how interaction based prioritization techniques perform, in terms of early fault detection when we prioritize based on 3-way interactions. We generalize prior work on prioritizing using 2-way interactions to t-way prioritization, and empirically evaluate this on three open source subjects, across multiple versions of each. We examine techniques that prioritize both existing CIT suites as well as generate new ones in prioritized order. We find that early fault detection can be improved when prioritizing 3-way CIT test suite by interactions that cover more code, and to a lesser degree when generating tests in prioritized order. Our techniques that work only from the specification, appear to work best with 2-way generation. I.
Combinatorial-based prioritization for usersession-based test suites
, 2012
"... Part of the Computer Sciences Commons This Thesis is brought to you for free and open access by the Graduate Studies at ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
Part of the Computer Sciences Commons This Thesis is brought to you for free and open access by the Graduate Studies at
