Results 1 - 10
of
21
How we get there: A context-guided search strategy in concolic testing.
- In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE
, 2014
"... ABSTRACT One of the biggest challenges in concolic testing, an automatic test generation technique, is its huge search space. Concolic testing generates next inputs by selecting branches from previous execution paths. However, a large number of candidate branches makes a simple exhaustive search in ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
ABSTRACT One of the biggest challenges in concolic testing, an automatic test generation technique, is its huge search space. Concolic testing generates next inputs by selecting branches from previous execution paths. However, a large number of candidate branches makes a simple exhaustive search infeasible, which often leads to poor test coverage. Several search strategies have been proposed to explore high-priority branches only. Each strategy applies different criteria to the branch selection process but most do not consider context, how we got to the branch, in the selection process. In this paper, we introduce a context-guided search (CGS) strategy. CGS looks at preceding branches in execution paths and selects a branch in a new context for the next input. We evaluate CGS with two publicly available concolic testing tools, CREST and CarFast, on six C subjects and six Java subjects. The experimental results show that CGS achieves the highest coverage of all twelve subjects and reaches a target coverage with a much smaller number of iterations on most subjects than other strategies.
Estimating Software Testing Complexity
"... Context: Complexity measures provide us some information about software artifacts. A measure of the difficulty of testing a piece of code could be very useful to take control about the test phase. Objective: The aim in this paper is the definition of a new measure of the difficulty for a computer to ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Context: Complexity measures provide us some information about software artifacts. A measure of the difficulty of testing a piece of code could be very useful to take control about the test phase. Objective: The aim in this paper is the definition of a new measure of the difficulty for a computer to generate test cases, we call it Branch Coverage Expectation (BCE). We also analyze the most common complexity measures and the most important features of a program. With this analysis we are trying to discover whether there exists a relationship between them and the code coverage of an automatically generated test suite. Method: The definition of this measure is based on a Markov model of the program. This model is used not only to compute the BCE, but also to provide an estimation of the number of test cases needed to reach a given coverage level in the program. In order to check our proposal, we perform a theoretical validation and we carry out an empirical validation study using 2600 test programs. Results: The results show that the previously existing measures are not so useful to estimate the difficulty of testing a program, because they are not highly correlated with the code coverage. Our proposed measure is much more correlated with the code coverage than the existing complexity measures. Conclusion: The high correlation of our measure with the code coverage suggests that the BCE measure is a very promising way of measuring the difficulty to automatically test a program. Our proposed measure is useful for predicting the behaviour of an automatic test case generator.
Quantifying the Characteristics of Java Programs that May Influence Symbolic Execution from a Test Data Generation Perspective
"... Abstract—Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faul ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Abstract—Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Manually creating test cases that satisfy these criteria is time consuming, error prone, and unwieldy. Symbolic execution has been used as an effective way of automatically generating test data that meets those criteria. Although this technique has been used for over three decades, several challenges remain, such as path explosion, precision of floating-point data, constraints with complex expressions, and dependency of external libraries. In this paper, we explore a sample of 100 open source Java projects in order to analyze characteristics that are relevant to generate test data using symbolic execution. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications. Keywords—Software testing; symbolic execution; test data gen-eration; I.
Synthesis of Geometry Proof Problems
"... This paper presents a semi-automated methodology for gen-erating geometric proof problems of the kind found in a high-school curriculum. We formalize the notion of a geometry proof problem and describe an algorithm for generating such problems over a user-provided figure. Our experimental re-sults i ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
This paper presents a semi-automated methodology for gen-erating geometric proof problems of the kind found in a high-school curriculum. We formalize the notion of a geometry proof problem and describe an algorithm for generating such problems over a user-provided figure. Our experimental re-sults indicate that our problem generation algorithm can ef-fectively generate proof problems in elementary geometry. On a corpus of 110 figures taken from popular geometry text-books, our system generated an average of about 443 prob-lems per figure in an average time of 4.7 seconds per figure. 1
Dodona: Automated Oracle Data Set Selection
"... Software complexity has increased the need for automated software testing. Most research on automating testing, how-ever, has focused on creating test input data. While careful selection of input data is necessary to reach faulty states in a system under test, test oracles are needed to actually det ..."
Abstract
- Add to MetaCart
(Show Context)
Software complexity has increased the need for automated software testing. Most research on automating testing, how-ever, has focused on creating test input data. While careful selection of input data is necessary to reach faulty states in a system under test, test oracles are needed to actually detect failures. In this work, we describe Dodona, a sys-tem that supports the generation of test oracles. Dodona ranks program variables based on the interactions and de-pendencies observed between them during program execu-tion. Using this ranking, Dodona proposes a set of variables to be monitored, that can be used by engineers to construct assertion-based oracles. Our empirical study of Dodona re-veals that it is more effective and efficient than the current state-of-the-art approach for generating oracle data sets, and can often yield oracles that are almost as effective as oracles hand-crafted by engineers without support.
Dodona: Automated Oracle Data Set Selection
"... Software complexity has increased the need for automated software testing. Most research on automating testing, how-ever, has focused on creating test input data. While careful selection of input data is necessary to reach faulty states in a system under test, test oracles are needed to actually det ..."
Abstract
- Add to MetaCart
(Show Context)
Software complexity has increased the need for automated software testing. Most research on automating testing, how-ever, has focused on creating test input data. While careful selection of input data is necessary to reach faulty states in a system under test, test oracles are needed to actually detect failures. In this work, we describe Dodona, a sys-tem that supports the generation of test oracles. Dodona ranks program variables based on the interactions and de-pendencies observed between them during program execu-tion. Using this ranking, Dodona proposes a set of variables to be monitored, that can be used by engineers to construct assertion-based oracles. Our empirical study of Dodona re-veals that it is more effective and efficient than the current state-of-the-art approach for generating oracle data sets, and can often yield oracles that are almost as effective as oracles hand-crafted by engineers without support. 1.
The ART of Divide and Conquer An Innovative Approach to Improving the Efficiency of Adaptive Random Testing
- THE SYMPOSIUM ON ENGINEERING TEST HARNESS (TSETH '13), IN PROCEEDINGS OF THE 13TH INTERNATIONAL CONFERENCE ON QUALITY SOFTWARE (QSIC 2013)
, 2013
"... Test case selection is a prime process in the engineering of test harnesses. In particular, test case diversity is an important concept. In order to achieve an even spread of test cases across the input domain, Adaptive Random Testing (ART) was proposed such that the history of previously executed ..."
Abstract
- Add to MetaCart
(Show Context)
Test case selection is a prime process in the engineering of test harnesses. In particular, test case diversity is an important concept. In order to achieve an even spread of test cases across the input domain, Adaptive Random Testing (ART) was proposed such that the history of previously executed test cases are taken into consideration when selecting the next test case. This was achieved through various means such as best candidate selection, exclusion, partitioning, and diversity metrics. Empirical studies showed that ART algorithms make good use of the concept of even spreading and achieve 40 to 50% improvement in test effectiveness over random testing in revealing the first failure, which is close to the theoretical limit. However, the computational complexity of ART algorithms may be quadratic or higher, and hence efficiency is an issue when a large number of previously executed test cases are involved. This paper proposes an innovative divide-and-conquer approach to improve the efficiency of ART algorithms while maintaining their performance in effectiveness. Simulation studies have been conducted to gauge its efficiency against two most commonly used ART algorithms, namely, fixed size candidate set and restricted random testing. Initial experimental results show that the divide-and-conquer technique can provide much better efficiency while maintaining similar, or even better, effectiveness.
An Automated Sequence Model Testing (ASMT) for Reducing Test Suit Size
"... Quality assertion of software is primarily done by means of testing an activity that faces constraints of both time & resources. Conventional testing strategies required the heavy resources & it can be done after the complete code is developed. So to apply some changes after the development ..."
Abstract
- Add to MetaCart
(Show Context)
Quality assertion of software is primarily done by means of testing an activity that faces constraints of both time & resources. Conventional testing strategies required the heavy resources & it can be done after the complete code is developed. So to apply some changes after the development requires lots of effort & cost. It also affects the time of deployment. So model based testing is a well accepted & dynamic approach for quality improvements because it provides affecting error detection at very low cost. It gives scalable & systematic solution to the test case reduction problem. To better understand the need of this early test case generation overall scenario of collaboration & development strategies is to be considered. The best way to get better knowledge of internal structure design models is the best option. Thus in this work we are focusing our research on path based testing by which internal structure of complete design can be verified & tested. Model based testing on path coverage is applied with standard coverage criteria for the large software’s & generates larger test counts. Our aim is to reduce those test counts with maximum coverage of testing at the early phases of SDLC even before the actual development starts. This paper proposes a novel automated sequence model based testing (ASMT) approach selects the test criteria based on UML diagram like activity, sequence, class etc. It applies the criteria with generation modules to create novel test cases. The typical deployment of ASMT goes through five stages: setting up test criteria, test model designing, test suite creation, performing test & analyzing the result. Thus by combining all the above methods improved test strategies can be designed. At the initial level of our research the approach seems to be better that others & will proves its effectiveness in future implementation.
Moving the Goalposts: Coverage Satisfaction Is Not Enough∗
"... Structural coverage criteria have been proposed to measure the ad-equacy of testing efforts. Indeed, in some domains—e.g., criti-cal systems areas—structural coverage criteria must be satisfied to achieve certification. The advent of powerful search-based test gen-eration tools has given us the abil ..."
Abstract
- Add to MetaCart
(Show Context)
Structural coverage criteria have been proposed to measure the ad-equacy of testing efforts. Indeed, in some domains—e.g., criti-cal systems areas—structural coverage criteria must be satisfied to achieve certification. The advent of powerful search-based test gen-eration tools has given us the ability to generate test inputs to satisfy these structural coverage criteria. While tempting, recent empiri-cal evidence indicates these tools should be used with caution, as merely achieving high structural coverage is not necessarily indica-tive of high fault detection ability. In this report, we review some of these findings, and offer recommendations on how the strengths of search-based test generation methods can alleviate these issues.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, TBD 1 The Risks of Coverage-Directed Test Case Generation
"... Abstract—A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation to ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation tools, it is tempting to simply generate test inputs to satisfy these structural coverage criteria. However, while techniques to produce coverage-providing tests are well established, the effectiveness of such approaches in terms of fault detection ability has not been adequately studied. In this work, we evaluate the effectiveness of test suites generated to satisfy four coverage criteria through counterexample-based test generation and a random generation approach—where tests are randomly generated until coverage is achieved—contrasted against purely random test suites of equal size. Our results yield three key conclusions. First, coverage criteria satisfaction alone can be a poor indication of fault finding effectiveness, with inconsistent results between the seven case examples (and random test suites of equal size often providing similar—or even higher—levels of fault finding). Second, the use of structural coverage as a supplement—rather than a target—for test generation can have a positive impact, with random test suites reduced to a coverage-providing subset detecting up to 13.5 % more faults than test suites generated specifically to achieve coverage. Finally, Observable MC/DC, a criterion designed to account for program structure and the selection of the test oracle, can—in part—address the failings of traditional structural coverage criteria, allowing for the generation of test suites achieving higher levels of fault detection than random test suites of equal size. These observations point to risks inherent in the increase in test automation in critical systems, and the need for more research in how coverage criteria, test generation approaches, the test oracle used, and system structure jointly influence test effectiveness. Index Terms—Software Testing, System Testing F
