Results 1 - 10
of
40
Timeaware test-case prioritization using integer linear programming
- in ISSTA
, 2009
"... Techniques for test-case prioritization re-order test cases to increase their rate of fault detection. When there is a fixed time budget that does not allow the execution of all the test cases, time-aware techniques for test-case prioritization may achieve a better rate of fault detection than tradi ..."
Abstract
-
Cited by 23 (6 self)
- Add to MetaCart
(Show Context)
Techniques for test-case prioritization re-order test cases to increase their rate of fault detection. When there is a fixed time budget that does not allow the execution of all the test cases, time-aware techniques for test-case prioritization may achieve a better rate of fault detection than traditional techniques for test-case prioritization. In this paper, we propose a novel approach to time-aware test-case prioritization using integer linear programming. To evaluate our approach, we performed experiments on two subject programs involving four techniques for our approach, two techniques for an approach to time-aware test-case prioritization based on genetic algorithms, and four traditional techniques for testcase prioritization. The empirical results indicate that two of our techniques outperform all the other techniques for the two subjects under the scenarios of both general and version-specific prioritization. The empirical results also indicate that some traditional techniques with lower analysis time cost for test-case prioritization may still perform competitively when the time budget is not quite tight.
An Orchestrated Survey on Automated Software Test Case Generation
, 2013
"... Test case generation is among the most labour-intensive tasks in software testing and also one that has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has also been one of the most active topics in the research on software testing for several decades, ..."
Abstract
-
Cited by 21 (2 self)
- Add to MetaCart
Test case generation is among the most labour-intensive tasks in software testing and also one that has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has also been one of the most active topics in the research on software testing for several decades, resulting in many different approaches and tools. This paper presents an orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in self-standing sections. The techniques presented include: (a) structural testing using symbolic execution, (b) model-based testing, (c) combinatorial testing, (d) random testing and its variant of adaptive random testing, and (e) search-based testing. Each section is contributed by world-renowned active researchers on the technique, and briefly covers the basic ideas underlying the technique, the current state of art, a discussion of the open research problems, and a perspective of the future development in the approach. As a whole, the paper aims at giving an introductory, up-to-date and (relatively) short overview of research in automatic test case generation, while ensuring comprehensiveness and authoritativeness.
GUI Interaction Testing: Incorporating Event Context
, 2011
"... Graphical user interfaces (GUIs), due to their event driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing it is important to “adequately cover ” this interaction space. In this paper, we develop a new family of coverage criteria for GU ..."
Abstract
-
Cited by 20 (5 self)
- Add to MetaCart
Graphical user interfaces (GUIs), due to their event driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing it is important to “adequately cover ” this interaction space. In this paper, we develop a new family of coverage criteria for GUI testing grounded in combinatorial interaction testing. The key motivation of using combinatorial techniques is that they enable us to incorporate “context” into the criteria in terms of event combinations, sequence length, and by including all possible positions for each event. Our new criteria range in both efficiency (measured by the size of the test suite) and effectiveness (the ability of the test suites to detect faults). In a case study on eight applications, we automatically generate test cases and systematically explore the impact of context, as captured by our new criteria. Our study shows that by increasing the event combinations tested and by controlling the relative positions of events defined by the new criteria, we can detect a large number of faults that were undetectable by earlier techniques.
Bridging the Gap between the Total and Additional Test-Case Prioritization Strategies
"... Abstract—In recent years, researchers have intensively investigated various topics in test-case prioritization, which aims to re-order test cases to increase the rate of fault detection during regression testing. The total and additional prioritization strategies, which prioritize based on total num ..."
Abstract
-
Cited by 19 (8 self)
- Add to MetaCart
(Show Context)
Abstract—In recent years, researchers have intensively investigated various topics in test-case prioritization, which aims to re-order test cases to increase the rate of fault detection during regression testing. The total and additional prioritization strategies, which prioritize based on total numbers of elements covered per test, and numbers of additional (not-yet-covered) elements covered per test, are two widely-adopted generic strategies used for such prioritization. This paper proposes a basic model and an extended model that unify the total strategy and the additional strategy. Our models yield a spectrum of generic strategies ranging between the total and additional strategies, depending on a parameter referred to as the p value. We also propose four heuristics to obtain differentiated p values for different methods under test. We performed an empirical study on 19 versions of four Java programs to explore our results. Our results demonstrate that wide ranges of strategies in our basic and extended models with uniform p values can significantly outperform both the total and additional strategies. In addition, our results also demonstrate that using differentiated p values for both the basic and extended models with method coverage can even outperform the additional strategy using statement coverage. I.
Swarm Testing
"... Abstract—Swarm testing is a novel and inexpensive way to improve the diversity of test cases generated during random testing. Increased diversity leads to improved coverage and fault detection. In swarm testing, the usual practice of potentially including all features in every test case is abandoned ..."
Abstract
-
Cited by 14 (9 self)
- Add to MetaCart
(Show Context)
Abstract—Swarm testing is a novel and inexpensive way to improve the diversity of test cases generated during random testing. Increased diversity leads to improved coverage and fault detection. In swarm testing, the usual practice of potentially including all features in every test case is abandoned. Rather, a large “swarm ” of randomly generated configurations, each of which omits some test features, is used, with configurations receiving equal resources. Omitting features can enable better exploration of the state space of a system. First, some features actively prevent the system from reaching interesting parts of its state space; e.g., “pop ” calls may prevent a stack data structure from executing a bug in its overflow detection logic. Second, even when there is no active suppression of behaviors, test features compete for space in each test, limiting the depth to which logic driven by features can be explored. Experimental results show that swarm testing increases coverage and can improve fault detection dramatically; for example, in a week of testing it found 42 % more distinct ways to crash a collection of C compilers than did the heavily hand-tuned default configuration of a random tester. Keywords-random testing; configuration diversity I.
Automated Diagnosis of Software Configuration Errors
"... Abstract—The behavior of a software system often depends on how that system is configured. Small configuration errors can lead to hard-to-diagnose undesired behaviors. We present a technique (and its tool implementation, called ConfDiagnoser) to identify the root cause of a configuration error — a s ..."
Abstract
-
Cited by 12 (2 self)
- Add to MetaCart
(Show Context)
Abstract—The behavior of a software system often depends on how that system is configured. Small configuration errors can lead to hard-to-diagnose undesired behaviors. We present a technique (and its tool implementation, called ConfDiagnoser) to identify the root cause of a configuration error — a single configuration option that can be changed to produce desired behavior. Our technique uses static analysis, dynamic profiling, and statistical analysis to link the undesired behavior to specific configuration options. It differs from existing approaches in two key aspects: it does not require users to provide a testing oracle (to check whether the software functions correctly) and thus is fully automated; and it can diagnose both crashing and noncrashing errors. We evaluated ConfDiagnoser on 5 non-crashing configuration errors and 9 crashing configuration errors from 5 configurable software systems written in Java. On average, the root cause was ConfDiagnoser’s fifth-ranked suggestion; in 10 out of 14 errors, the root cause was one of the top 3 suggestions; and more than half of the time, the root cause was the first suggestion. I.
Feature Interaction Faults Revisited: An Exploratory Study
- In Proc. of ISSRE
, 2011
"... Abstract—While a large body of research is dedicated to testing for feature interactions in configurable software, there has been little work that examines what constitutes such a fault at the code level. In consequence, we do not know how prevalent real interaction faults are in practice, what a ty ..."
Abstract
-
Cited by 11 (3 self)
- Add to MetaCart
(Show Context)
Abstract—While a large body of research is dedicated to testing for feature interactions in configurable software, there has been little work that examines what constitutes such a fault at the code level. In consequence, we do not know how prevalent real interaction faults are in practice, what a typical interaction fault looks like in code, how to seed interaction faults, or whether current interaction testing techniques are effective at finding the faults they aim to detect. We make a first step in this direction, by deriving a whitebox criterion for an interaction fault. Armed with this criterion, we perform an exploratory study on hundreds of faults from the field in two open source systems. We find that only three of the 28 which appear to be interaction faults are in fact due to features ’ interactions. We investigate the remaining 25 and find that, although they could have been detected without interaction testing, varying the system configuration amplifies the fault-finding power of a test suite, making these faults easier to expose. Thus, we characterize the benefits of interaction testing in regards to both interaction and non-interaction faults. We end with a discussion of several mutations that can be used to mimic interaction faults based on the faults we see in practice.
Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing
"... Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. However, in order to be practically applicable, it must be able to cater for soft and hard real-world constraints ..."
Abstract
-
Cited by 9 (4 self)
- Add to MetaCart
(Show Context)
Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. However, in order to be practically applicable, it must be able to cater for soft and hard real-world constraints and should, ideally, report a test priority order that maximises earliest fault detection. We show that we can achieve higher strength CIT (previously thought infeasible). Furthermore, we show that higher strength suites find more faults, while prioritisation using lower strengths are no worse at achieving early fault revelation. Our constrained 5-way interaction suites can be automatically constructed in 6 minutes on average (reduced from p hours for their unconstrained counterparts). These test suites find 4.22 % more faults than traditional pairwise suites. 1.
Improving the Testing and Testability of Software Product Lines
- IN PROC. SPLC, LNCS 6287
, 2010
"... Software Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, for better understanding of testability issues and for testing techniques that can operate cost-effectively on SPLs. In this paper we consider these ..."
Abstract
-
Cited by 9 (0 self)
- Add to MetaCart
(Show Context)
Software Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, for better understanding of testability issues and for testing techniques that can operate cost-effectively on SPLs. In this paper we consider these testability issues and highlight some differences between optional versus alternative features. We then provide a graph based testing approach called the FIG Basis Path method that selects products and features for testing based on a feature dependency graph. We conduct a case study on several non-trivial SPLs and show that for these subjects, the FIG Basis Path method is as effective as testing all products, but tests no more than 24 % of the products in the SPL. 1
Using Feature Locality: Can We Leverage History to Avoid Failures During Reconfiguration?
"... Despite the best efforts of software engineers, faults still escape into deployed software. Developers need time to prepare and distribute fixes, and in the interim deployments must either tolerate or avoid failures. Self-adaptive systems, systems that adapt to meet changing requirements in a dynami ..."
Abstract
-
Cited by 7 (1 self)
- Add to MetaCart
(Show Context)
Despite the best efforts of software engineers, faults still escape into deployed software. Developers need time to prepare and distribute fixes, and in the interim deployments must either tolerate or avoid failures. Self-adaptive systems, systems that adapt to meet changing requirements in a dynamic environment, have a daunting task if their reconfiguration involves adding or removing functional features, because configurable software is known to suffer from failures that appear only under certain feature combinations. Although configuration-dependent failures may be difficult to provoke, and thus hard to detect in testing, we posit that they also constitute opportunities for reconfiguration to increase system reliability. We further conjecture that the failures that are sensitive to a system configuration depend on similar feature combinations, a phenomenon we call feature-locality, and that this locality can be combined with historical data to predict failure-prone configurations. In a case study on 128 failures reported against released versions of an open source configurable system, we find evidence to support our hypothesis. We show that only a small number of features affect the visibility of these failures, and that over time we can learn these features to avoid future failures.
