Results 1 - 10
of
20
Estimating Software Testing Complexity
"... Context: Complexity measures provide us some information about software artifacts. A measure of the difficulty of testing a piece of code could be very useful to take control about the test phase. Objective: The aim in this paper is the definition of a new measure of the difficulty for a computer to ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Context: Complexity measures provide us some information about software artifacts. A measure of the difficulty of testing a piece of code could be very useful to take control about the test phase. Objective: The aim in this paper is the definition of a new measure of the difficulty for a computer to generate test cases, we call it Branch Coverage Expectation (BCE). We also analyze the most common complexity measures and the most important features of a program. With this analysis we are trying to discover whether there exists a relationship between them and the code coverage of an automatically generated test suite. Method: The definition of this measure is based on a Markov model of the program. This model is used not only to compute the BCE, but also to provide an estimation of the number of test cases needed to reach a given coverage level in the program. In order to check our proposal, we perform a theoretical validation and we carry out an empirical validation study using 2600 test programs. Results: The results show that the previously existing measures are not so useful to estimate the difficulty of testing a program, because they are not highly correlated with the code coverage. Our proposed measure is much more correlated with the code coverage than the existing complexity measures. Conclusion: The high correlation of our measure with the code coverage suggests that the BCE measure is a very promising way of measuring the difficulty to automatically test a program. Our proposed measure is useful for predicting the behaviour of an automatic test case generator.
Quantifying the Characteristics of Java Programs that May Influence Symbolic Execution from a Test Data Generation Perspective
"... Abstract—Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faul ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Abstract—Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Manually creating test cases that satisfy these criteria is time consuming, error prone, and unwieldy. Symbolic execution has been used as an effective way of automatically generating test data that meets those criteria. Although this technique has been used for over three decades, several challenges remain, such as path explosion, precision of floating-point data, constraints with complex expressions, and dependency of external libraries. In this paper, we explore a sample of 100 open source Java projects in order to analyze characteristics that are relevant to generate test data using symbolic execution. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications. Keywords—Software testing; symbolic execution; test data gen-eration; I.
Synthesis of Geometry Proof Problems
"... This paper presents a semi-automated methodology for gen-erating geometric proof problems of the kind found in a high-school curriculum. We formalize the notion of a geometry proof problem and describe an algorithm for generating such problems over a user-provided figure. Our experimental re-sults i ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
This paper presents a semi-automated methodology for gen-erating geometric proof problems of the kind found in a high-school curriculum. We formalize the notion of a geometry proof problem and describe an algorithm for generating such problems over a user-provided figure. Our experimental re-sults indicate that our problem generation algorithm can ef-fectively generate proof problems in elementary geometry. On a corpus of 110 figures taken from popular geometry text-books, our system generated an average of about 443 prob-lems per figure in an average time of 4.7 seconds per figure. 1
An Automated Sequence Model Testing (ASMT) for Reducing Test Suit Size
"... Quality assertion of software is primarily done by means of testing an activity that faces constraints of both time & resources. Conventional testing strategies required the heavy resources & it can be done after the complete code is developed. So to apply some changes after the development ..."
Abstract
- Add to MetaCart
(Show Context)
Quality assertion of software is primarily done by means of testing an activity that faces constraints of both time & resources. Conventional testing strategies required the heavy resources & it can be done after the complete code is developed. So to apply some changes after the development requires lots of effort & cost. It also affects the time of deployment. So model based testing is a well accepted & dynamic approach for quality improvements because it provides affecting error detection at very low cost. It gives scalable & systematic solution to the test case reduction problem. To better understand the need of this early test case generation overall scenario of collaboration & development strategies is to be considered. The best way to get better knowledge of internal structure design models is the best option. Thus in this work we are focusing our research on path based testing by which internal structure of complete design can be verified & tested. Model based testing on path coverage is applied with standard coverage criteria for the large software’s & generates larger test counts. Our aim is to reduce those test counts with maximum coverage of testing at the early phases of SDLC even before the actual development starts. This paper proposes a novel automated sequence model based testing (ASMT) approach selects the test criteria based on UML diagram like activity, sequence, class etc. It applies the criteria with generation modules to create novel test cases. The typical deployment of ASMT goes through five stages: setting up test criteria, test model designing, test suite creation, performing test & analyzing the result. Thus by combining all the above methods improved test strategies can be designed. At the initial level of our research the approach seems to be better that others & will proves its effectiveness in future implementation.
Entropy-Based Test Generation for Improved Fault Localization
"... Abstract—Spectrum-based Bayesian reasoning can effectively rank candidate fault locations based on passing/failing test cases, but the diagnostic quality highly depends on the size and diversity of the underlying test suite. As test suites in practice often do not exhibit the necessary properties, w ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—Spectrum-based Bayesian reasoning can effectively rank candidate fault locations based on passing/failing test cases, but the diagnostic quality highly depends on the size and diversity of the underlying test suite. As test suites in practice often do not exhibit the necessary properties, we present a technique to extend existing test suites with new test cases that optimize the diagnostic quality. We apply probability theory concepts to guide test case generation using entropy, such that the amount of uncertainty in the diagnostic ranking is minimized. Our ENTBUG prototype extends the search-based test generation tool EVOSUITE to use entropy in the fitness function of its underlying genetic algorithm, and we applied it to seven real faults. Empirical results show that our approach reduces the entropy of the diagnostic ranking by 49 % on average (compared to using the original test suite), leading to a 91 % average reduction of diagnosis candidates needed to inspect to find the true faulty one. Index Terms—Fault localization; Test case generation. I.
string validation routines using web searches
"... Automatic generation of valid and invalid test data for ..."
(Show Context)
OF THE UNIVERSITY OF MINNESOTA BY
, 2015
"... Many, many people were pivitol in the creation of this thesis, and I would like to take a mo-ment to thank them (and numerous other people that I have inevitably forgotten to mention). I would like to thank Mats Heimdahl—mv advisor—whose support and guidance made this entire project possible. I’d li ..."
Abstract
- Add to MetaCart
(Show Context)
Many, many people were pivitol in the creation of this thesis, and I would like to take a mo-ment to thank them (and numerous other people that I have inevitably forgotten to mention). I would like to thank Mats Heimdahl—mv advisor—whose support and guidance made this entire project possible. I’d like to thank him in equal measure for telling me when I made a bone-headed mistake and for trusting me to find the method in my madness. I could not have asked for a better mentor. Thanks also go to Mike Whalen and Sanjai Rayadurgam, who de-voted their time, experience, and expertise to providing feedback on my ideas. I would also like to thank Eric Van Wyk and Galin Jones for serving on my thesis committee and providing excellent advice over the past few years. Research never takes place in a vacuum, and I would like to thank Matt Staats, Ian De Silva, Lian Duan, Hung Pham, Anitha Murugesan, Dongjiang You, Jason Biatek, Kevin Wendt, and the rest of the Crisys crew for being both great coworkers and great friends. I am grateful to Tim Menzies, Mark Harman, Phil McMinn, and Misty Davies for providing opportunities to collaborate on research and take a leading role in the software engineering research community. Last—but certainly not least—I would like to thank my wife, my friends, and my family for their neverending work in keeping me sane. Michelle, you’re a saint.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, TBD 1 The Risks of Coverage-Directed Test Case Generation
"... Abstract—A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation to ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation tools, it is tempting to simply generate test inputs to satisfy these structural coverage criteria. However, while techniques to produce coverage-providing tests are well established, the effectiveness of such approaches in terms of fault detection ability has not been adequately studied. In this work, we evaluate the effectiveness of test suites generated to satisfy four coverage criteria through counterexample-based test generation and a random generation approach—where tests are randomly generated until coverage is achieved—contrasted against purely random test suites of equal size. Our results yield three key conclusions. First, coverage criteria satisfaction alone can be a poor indication of fault finding effectiveness, with inconsistent results between the seven case examples (and random test suites of equal size often providing similar—or even higher—levels of fault finding). Second, the use of structural coverage as a supplement—rather than a target—for test generation can have a positive impact, with random test suites reduced to a coverage-providing subset detecting up to 13.5 % more faults than test suites generated specifically to achieve coverage. Finally, Observable MC/DC, a criterion designed to account for program structure and the selection of the test oracle, can—in part—address the failings of traditional structural coverage criteria, allowing for the generation of test suites achieving higher levels of fault detection than random test suites of equal size. These observations point to risks inherent in the increase in test automation in critical systems, and the need for more research in how coverage criteria, test generation approaches, the test oracle used, and system structure jointly influence test effectiveness. Index Terms—Software Testing, System Testing F
Moving the Goalposts: Coverage Satisfaction Is Not Enough∗
"... Structural coverage criteria have been proposed to measure the ad-equacy of testing efforts. Indeed, in some domains—e.g., criti-cal systems areas—structural coverage criteria must be satisfied to achieve certification. The advent of powerful search-based test gen-eration tools has given us the abil ..."
Abstract
- Add to MetaCart
(Show Context)
Structural coverage criteria have been proposed to measure the ad-equacy of testing efforts. Indeed, in some domains—e.g., criti-cal systems areas—structural coverage criteria must be satisfied to achieve certification. The advent of powerful search-based test gen-eration tools has given us the ability to generate test inputs to satisfy these structural coverage criteria. While tempting, recent empiri-cal evidence indicates these tools should be used with caution, as merely achieving high structural coverage is not necessarily indica-tive of high fault detection ability. In this report, we review some of these findings, and offer recommendations on how the strengths of search-based test generation methods can alleviate these issues.
An Automated Sequence Model Testing (ASMT) For Improved Test Case Generation Using Cloud Integration
"... Abstract- Software quality assurance is primarily done by means of testing an activity that faces constraints of both time & resources. Traditional testing strategies obliged the overwhelming resources and infrastructure and it can be carried out after the completion code is created. So to apply ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract- Software quality assurance is primarily done by means of testing an activity that faces constraints of both time & resources. Traditional testing strategies obliged the overwhelming resources and infrastructure and it can be carried out after the completion code is created. So to apply some changes after the development obliges loads of effort & cost. It likewise affects the time of deployment. So model based testing is a generally accepted & dynamic approach for quality improvements because it gives affecting mistake detection easily. It gives scalable & systematic solution to the test case reduction problem. To better understand the need of this early test case generation general scenario of collaboration & development strategies is to be considered. The most ideal approach to improve learning of internal structure design models is the best choice. In this manner, in
