Results 1 - 10
of
17
Directed test generation for effective fault localization
- In Proceedings of the 19th international symposium on Software testing and analysis, ISSTA ’10
"... ABSTRACT Fault-localization techniques that apply statistical analyses to execution data gathered from multiple tests are quite effective when a large test suite is available. However, if no test suite is available, what is the best approach to generate one? This paper investigates the fault-locali ..."
Abstract
-
Cited by 35 (0 self)
- Add to MetaCart
(Show Context)
ABSTRACT Fault-localization techniques that apply statistical analyses to execution data gathered from multiple tests are quite effective when a large test suite is available. However, if no test suite is available, what is the best approach to generate one? This paper investigates the fault-localization effectiveness of test suites generated according to several test-generation techniques based on combined concrete and symbolic (concolic) execution. We evaluate these techniques by applying the Ochiai fault-localization technique to generated test suites in order to localize 35 faults in four PHP Web applications. Our results show that the test-generation techniques under consideration produce test suites with similar high fault-localization effectiveness, when given a large time budget. However, a new, "directed" test-generation technique, which aims to maximize the similarity between the path constraints of the generated tests and those of faulty executions, reaches this level of effectiveness with much smaller test suites. On average, when compared to test generation based on standard concolic execution techniques that aims to maximize code coverage, the new directed technique preserves faultlocalization effectiveness while reducing test-suite size by 86.1% and test-suite generation time by 88.6%.
Fault Localization through Evaluation Sequences
- JOURNAL OF SYSTEMS AND SOFTWARE
, 2010
"... Predicate-based statistical fault-localization techniques find fault-relevant predicates in a program by contrasting the statistics of the evaluation results of individual predicates between failed runs and successful runs. While short-circuit evaluations may occur in program executions, treating pr ..."
Abstract
-
Cited by 11 (6 self)
- Add to MetaCart
Predicate-based statistical fault-localization techniques find fault-relevant predicates in a program by contrasting the statistics of the evaluation results of individual predicates between failed runs and successful runs. While short-circuit evaluations may occur in program executions, treating predicates as atomic units ignores this fact, masking out various types of useful statistics on dynamic program behavior. In this paper, we differentiate the short-circuit evaluations of individual predicates on individual program statements, producing one set of evaluation sequences per predicate. We then investigate experimentally the effectiveness of using these sequences to locate faults by comparing existing predicate-based techniques with and without such differentiation. We use both the Siemens program suite and four real-life UNIX utility programs as our subjects. The experimental results show that the proposed use of short-circuit evaluations can, on average, improve predicate-based statistical fault-localization techniques while incurring relatively small performance overhead.
A dynamic test cluster sampling strategy by leveraging execution spectra information
- In Proceedings of Conference on Software Testing, Verification and Validation (ICST’10
"... Abstract—Cluster filtering is a kind of test selection technique, which saves human efforts for result inspection by reducing test size and finding maximum failures. Cluster sampling strategies play a key role in the cluster filtering technique. A good sampling strategy can greatly improve the failu ..."
Abstract
-
Cited by 9 (5 self)
- Add to MetaCart
(Show Context)
Abstract—Cluster filtering is a kind of test selection technique, which saves human efforts for result inspection by reducing test size and finding maximum failures. Cluster sampling strategies play a key role in the cluster filtering technique. A good sampling strategy can greatly improve the failure detection capability. In this paper, we propose a new cluster sampling strategy called execution-spectra-based sampling (ESBS). Different from the existing sampling strategies, ESBS iteratively selects test cases from each cluster. In each iteration process, ESBS selects the test case that has the maximum suspiciousness to be a failed test. For each test, its suspiciousness is computed based on the execution spectra information of previous passed and failed test cases selected from the same cluster. The new sampling strategy ESBS is evaluated experimentally and the results show that it is more effective than the existing sampling strategies in most cases. Keywords-test selection; cluster filtering; cluster sampling; execution spectra; I.
Non-Parametric Statistical Fault Localization
- JOURNAL OF SYSTEMS AND SOFTWARE
, 2011
"... Fault localization is a major activity in program debugging. To automate this time-consuming task, many existing fault-localization techniques compare passed executions and failed executions, and suggest suspicious program elements, such as predicates or statements, to facilitate the identification ..."
Abstract
-
Cited by 9 (4 self)
- Add to MetaCart
Fault localization is a major activity in program debugging. To automate this time-consuming task, many existing fault-localization techniques compare passed executions and failed executions, and suggest suspicious program elements, such as predicates or statements, to facilitate the identification of faults. To do that, these techniques propose statistical models and use hypothesis testing methods to test the similarity or dissimilarity of proposed program features between passed and failed executions. Furthermore, when applying their models, these techniques presume that the feature spectra come from populations with specific distributions. The accuracy of using a model to describe feature spectra is related to and may be affected by the underlying distribution of the feature spectra, and the use of a (sound) model on inapplicable circumstances to describe real-life feature spectra may lower the effectiveness of these fault-localization techniques. In this paper, we make use of hypothesis testing methods as the core concept in developing a predicate-based fault-localization framework. We report a controlled experiment to compare, within our framework, the efficacy, scalability, and efficiency of applying three categories of hypothesis testing methods, namely, standard non-parametric hypothesis testing methods, standard parametric hypothesis testing methods, and debugging-specific parametric testing methods. We also conduct a case study to compare the effectiveness of the winner of these three categories with the effectiveness of 33 existing statement-level fault-localization techniques. The experimental results show that the use of non-parametric hypothesis testing methods in our proposed predicate-based fault-localization model is the most promising.
Is Non-Parametric Hypothesis Testing Model Robust for Statistical Fault Localization?
- INFORMATION AND SOFTWARE TECHNOLOGY
, 2009
"... Fault localization is the most difficult activity in software debugging. Many existing statistical fault-localization techniques estimate the fault positions of programs by comparing the program feature spectra between passed runs and failed runs. Some existing approaches develop estimation formulas ..."
Abstract
-
Cited by 8 (4 self)
- Add to MetaCart
Fault localization is the most difficult activity in software debugging. Many existing statistical fault-localization techniques estimate the fault positions of programs by comparing the program feature spectra between passed runs and failed runs. Some existing approaches develop estimation formulas based on mean values of the underlying program feature spectra and their distributions alike. Our previous work advocates the use of a non-parametric approach in estimation formulas to pinpoint fault-relevant positions. It is worthy of further study to resolve the two schools of thought by examining the fundamental, underlying properties of distributions related to fault localization. In particular, we ask: Can the feature spectra of program elements be safely considered as normal distributions so that parametric techniques can be soundly and powerfully applied? In this paper, we empirically investigate this question from the program predicate perspective. We conduct an experimental study based on the Siemens suite of programs. We examine the degree of normality on the distributions of evaluation biases of the predicates, and obtain three major results from the study. First, almost all examined distributions of evaluation biases are either normal or far from normal, but not in between. Second, the most fault-relevant predicates are less likely to exhibit normal distributions in terms of evaluation biases than other predicates. Our results show that normality is not common as far as evaluation bias can represent. Furthermore, the effectiveness of our non-parametric predicate-based fault-localization technique weakly correlates with the distributions of evaluation biases, making the technique robust to this type of uncertainty in the underlying program spectra.
How Well Does Test Case Prioritization Integrate with Statistical Fault Localization?
- INFORMATION AND SOFTWARE TECHNOLOGY
, 2012
"... Context: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization.
Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prior ..."
Abstract
-
Cited by 7 (4 self)
- Add to MetaCart
Context: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization.
Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques.
Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and 4 statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, zed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed.
Result: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects.
Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging.
Localizing failure-inducing program edits based on spectrum information
- In ICSM
, 2011
"... Abstract—Keeping evolving systems fault free is hard. Change impact analysis is a well-studied methodology for finding faults in evolving systems. For example, in order to help developers identify failure-inducing edits, Chianti extracts program edits as atomic changes between different program vers ..."
Abstract
-
Cited by 6 (3 self)
- Add to MetaCart
(Show Context)
Abstract—Keeping evolving systems fault free is hard. Change impact analysis is a well-studied methodology for finding faults in evolving systems. For example, in order to help developers identify failure-inducing edits, Chianti extracts program edits as atomic changes between different program versions, selects affected tests, and determines a subset of those changes that might induce test failures. However, identifying real regression faults is challenging for developers since the number of affecting changes related to each test failure may still be too large for manual inspection. This paper presents a novel approach FAULTTRACER which ranks program edits in order to reduce developers’ effort in manually inspecting all affecting changes. FAULTTRACER adapts spectrum-based fault localization techniques and applies them in tandem with an enhanced change impact analysis that uses Extended Call Graphs to identify failure-inducing edits more precisely. We evaluate FAULTTRACER using 23 versions of 4 real-world Java programs from the Software Infrastructure Repository. The experimental results show that FAULTTRACER outperforms Chianti in selecting affected tests (slightly better, but handles safety problems of Chianti) as well as in determining affecting changes (with an improvement of approximately 20%). By ranking the affecting changes using spectrum-based test behavior profile, for 14 out of 22 studied failures, FAULTTRACER places a real regression fault within top 3 atomic changes, significantly reducing developers ’ effort in inspecting potential failure-inducing edits. I.
On the adoption of MC/DC and control-flow adequacy for a tight integration of program testing and statistical fault localization
- INFORMATION AND SOFTWARE TECHNOLOGY
, 2013
"... Context: Testing and debugging consume a significant portion of software development effort. Both processes are usually conducted independently despite their close relationship with each other. Test adequacy is vital for developers to assure that sufficient testing effort has been made, while findin ..."
Abstract
-
Cited by 2 (1 self)
- Add to MetaCart
Context: Testing and debugging consume a significant portion of software development effort. Both processes are usually conducted independently despite their close relationship with each other. Test adequacy is vital for developers to assure that sufficient testing effort has been made, while finding all the faults in a program as soon as possible is equally important. A tight integration between testing and debugging activities is essential. Objective: The paper aims at finding whether three factors, namely, the adequacy criterion to gauge a test suite, the size of a prioritized test suite, and the percentage of such a test suite used in fault localization, have significant impacts on integrating test case prioritization techniques with statistical fault localization techniques. Method: We conduct a controlled experiment to investigate the effectiveness of applying adequate test suites to locate faults in a benchmark suite of seven Siemens programs and four real-life UNIX utility programs using three adequacy criteria, 16 test case prioritization techniques, and four statistical fault localization techniques. We measure the proportion of code needed to be examined in order to locate a fault as the effectiveness of statistical fault localization techniques. We also investigate the integration of test case prioritization and statistical fault localization with postmortem analysis. Result: The main result shows that on average, it is more effective for a statistical fault localization technique to utilize the execution results of a MC/DC-adequate test suite than those of a branch-adequate test suite, and is in turn more effective to utilize the execution results of a branch-adequate test suite than those of a statement-adequate test suite. On the other hand,
Rights Creative Commons: Attribution 3.0 Hong Kong License To appear in Journal of Systems and Software Fault Localization through Evaluation Sequences I,II
"... Predicate-based statistical fault-localization techniques find fault-relevant predicates in a program by contrasting the statistics of the evaluation results of individual predicates between failed runs and successful runs. While short-circuit evaluations may occur in program executions, treating pr ..."
Abstract
- Add to MetaCart
Predicate-based statistical fault-localization techniques find fault-relevant predicates in a program by contrasting the statistics of the evaluation results of individual predicates between failed runs and successful runs. While short-circuit evaluations may occur in program executions, treating predicates as atomic units ignores this fact, masking out various types of useful statistics on dynamic program behavior. In this paper, we differentiate the short-circuit evaluations of individual predicates on individual program statements, producing one set of evaluation sequences per predicate. We then investigate experimentally the effectiveness of using these sequences to locate faults by comparing existing predicate-based techniques with and without such differentiation. We use both the Siemens program suite and four real-life UNIX utility programs as our subjects. The experimental results show that the proposed use of short-circuit evaluations can, on average, improve predicate-based statistical fault-localization techniques while incurring relatively small performance overhead. Key words: fault localization, Boolean expression, predicate, evaluation sequence I c © 2009 Elsevier Inc. This material is presented to ensure timely dissemination of scholarly and technical work.
of Spectrum-Based Fault Localization
, 2016
"... All in-text references underlined in blue are linked to publications on ResearchGate, letting you access and read them immediately. ..."
Abstract
- Add to MetaCart
(Show Context)
All in-text references underlined in blue are linked to publications on ResearchGate, letting you access and read them immediately.
