Horgan, J., London, S., Lyu, M., "Achieving Software Quality with Testing Coverage Measures", IEEE Computer, Vol. 27, No. 9, September 1994, pp. 60-69.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....re introduces JBlanket to find out if programmers are willing to work towards increasing coverage back up to 100 . 6.2.5 XC and System Quality In 1994 Horgan et. al presented two case studies in which they measured dataflow coverage using ATAC (Automated Test Analysis for v w ) on C programs [42]. ATAC measures block 71 coverage, decision coverage, c use (computational expression) p use (predicate) and all uses (either c use or p use) The first case study conducted in the large attempted to find a relationship between coverage measurement and the total faults by inspecting one of ....

....until eventually leveling out. Reaching above 80 percent coverage was an important step toward software quality and reliability. There did not seem be a strong correlation between the total faults detected in the program versions and their coverage measures during various testing conditions [42]. Since the study conducted by Horgan et. al did not measure method coverage, a study similar to this one could be conducted using XC. Begin with two systems in which one has access to JBlanket and the other does not. Require the system with JBlanket maintain 100 coverage throughout development. ....

Joseph R. Horgan, Saul London, and Michael R. Lyu. Achieving Software Quality with Testing Coverage Measures. Computer, 27:60--69, September 1994.

....able to reach 100 method level coverage. However, only 5 of the 8 services reached the 100 coverage at the end of this study. Two services missed complete extreme coverage by 1 method. Figure 3. LOC for CREST services. Figure 4. JBlanket coverage for CREST services. Similar to the findings in [14] that measured block coverage, decision (branch) coverage, c use (computational expression use) coverage, and p use (predicate use) coverage on C programs, the levels of coverage obtained by each service were not always consistent. For example, the coverage for the Resume service initially ....

Horgan, Joseph, London, Saul and Bellcore, Michael. Achieving Software Quality with Testing Coverage Measures. In Computer, pages 60-69, September 1994.

....are explicitly tested unless implementation constraints are part of the specification. Full code coverage testing allows a component to be rated at Level 4. Code coverage is relatively easy to achieve and there are a variety of tools which will provide coverage metrics of varying accuracy [Horgan et al. 1994, Lyu et al. 1994, Ltd, 2000] it was a straightforward exercise to extract coverage measurements from our symbolic execution system [Bundell et al. 2000b] Although easy to implement and measure, code coverage is known to have significant weaknesses if it is the sole test strategy. It is easy ....

....be readily determined to be dead or not. Statement coverage testing is clearly a necessary condition for complete testing, but it is easily shown to be insufficient. Its major advantage is that it is relatively easy to measure automatically: a variety of tools have been built for example, ATAC [Horgan et al. 1994] and C Cover [Ltd, 2000] be cause it is a reasonably simple task: we easily extended our symbolic execution system to measure statement coverage [Lee, 2000] Ensuring that all statements are executed allows a component to be rated at Level 4. However, note that statement coverage testing may ....

Horgan, J. R., London, S., and Lyu, M. R. (1994). Achieving software quality with testing coverage measures. IEEE Computer, 27(9):60-69.

....and create the framework for an enhanced software reliability model capable of estimating accurately the reliability of software products utilizing appropriately obtained test data. 2. 1 Existing Notions of Coverage The importance of test coverage has been recognized by several researchers [6, 8], and empirical evidence strongly suggests that testing which is carried out without some form of test coverage measurement may fail to sensitize as much as 45 of the code. Software designers and testers must develop effective evaluation tools capable of measuring test coverage and pointing out ....

....Such tools should also provide data which lead to improvements in the quality and effectiveness of a software test as well as information about its adequacy when deciding product release times. There are two types of coverage definitions in literature: control flow and data flow coverage[2, 6, 7, 16]. Each coverage criterion proposed in the literature captures some important aspect of a program s structure. In general, test coverage is a measure of how well a test covers all the potential faultsites in a software product under test. It should be obvious that how one defines a potential ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure," IEEE Computer, pp. 60-69, Sept. 1994.

....requires that all basic blocks are executed at least once for the chosen input sequence, otherwise additional input sequences must be chosen for the measurements. Identifying the input sequence(s) that uses all basic blocks can be performed through test coverage measures such as block coverage [9]. n p can be estimated as the sum of the estimated number of activated faults for all the n basic blocks as: # = n j j p j j p x x n n 1 , 3.5) Where n j is the number of faults activated during execution of block j for the base input sequence, x p, j is the number of ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving software quality with testing coverage measures", IEEE Computer, vol. 27, pp. 60-69, Sept. 1994.

....and create the framework for an enhanced software reliability model capable of estimating accurately the reliability of software products utilizing appropriately obtained test data. 2. 1 Existing Notions of Coverage The importance of test coverage has been recognized by several researchers [6, 8], and empirical evidence strongly suggests that testing which is carried out without some form of test coverage measurement may fail to sensitize as much as 45 of the code. Software designers and testers must develop effective evaluation tools capable of measuring test coverage and pointing out ....

....Such tools should also provide data which lead to improvements in the quality and effectiveness of a software test as well as information about its adequacy when deciding product release times. There are two types of coverage definitions in literature: control flow and data flow coverage[2, 6, 7, 16]. Each coverage criterion proposed in the literature captures some important aspect of a program s structure. In general, test coverage is a measure of how well a test covers all the potential faultsites in a software product under test. It should be obvious that how one defines a potential ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure," IEEE Computer, pp. 60-69, Sept. 1994.

.... introduce into the area of software reliability a number of well developed analytical techniques and notions which have been successfully applied to reliability studies in hardware systems[13] Experimental data such as those presented in Table 1 can be easily collected, using tools such as ATAC[4], and utilized to obtain the test coverage function, c(t) by associating coverage information given in column 2 with the cumulative execution time given in column 5. 2.2 Existing Software Reliability Models All the leading software reliability models which are widely used for software quality ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure, " IEEE Computer, vol. 27, no. 9, pp. 60-69, September 1994.

....into the program and the tester has to find the fault while minimizing the number of tests. A typical scenario in software testing is that the user tests his program by finding a test bucket (a set of inputs) that meets a certain coverage criteria, e.g. branch coverage or statement coverage [1, 2, 3]. It is plausible that in certain situations it might be possible to embed such a set of tests (e.g. the union over all test buckets that meet branch coverage) in a Poset or in a Tree such that the requirement for covering all tests can be replaced by a requirement for searching in this Poset or ....

Joseph R. Horgan, Saul London, and Michael R. Lyu. Achieving software quality with testing coverage measures. IEEE Transaction on Software Engineering, October 1993.

....measuring syntactic properties in the execution, for example, that each statement was executed, or each branch was taken. This makes program based coverage a generic method which is usually easy to measure, and for which many tools are available. Examples include program based coverage tools for C [9], C [13] and Java [14] Functional coverage, on the other hand, focuses on the functionality of the program, and it is used to check that every aspect of the functionality is tested. Therefore, functional coverage is design and implementation specific, and is more costly to measure. Currently, ....

J.R. Horgan, S. London and M.R. Lyu. Achieving software quality with testing coverage measures, Computer, 27(9):60-69, September 1994.

....Empirical studies. Case studies comparing software development methods are difficult to conduct. Attempts to establish a correlation between the degree of debug testing (usually measured by some structural coverage of unit tests) and the resulting software quality are at best preliminary [7, 13, 20, 9, 16]. On the other side, case studies using formal methods development show great variation, both in the care with which the method is defined and applied and in the results [11] Neither side has any real claim to establishing its case. Analysis of partition testing. A number of careful ....

J. Horgan, S. London, and M. Lyu. Achieving software quality with testing coverage. IEEE Computer, 27:60--69, 1994.

....that any solution be usable in typical testing conditions. The main focus of this paper is to present an easily replicable procedure that is based on simulation techniques and is designed to investigate the relationship between test coverage and defect coverage. Data coming from Horgan et al. [13] are used to exemplify the procedure and show how it can yield more precise results. The structure of the paper is as follows. We first give a precise definition of the problems associated with using test coverage measures for controlling test effectiveness. Then, in Section 3, we present a ....

....than a, then the null hypothesis cannot be rejected. In this case, we cannot conclude that test coverage has a significant impact on defect coverage. 4 Case Study Based on data that was generated during an experiment conducted by the University of Iowa and the Rockwell Collins Avionics Division [13], this section illustrates how to apply the statistical testing procedure defined in Section 3.2. 4.1 Background information The purpose of the experiment was to investigate the relationship between the coverage of program constructs during testing and defect coverage. For this purpose, based on ....

[Article contains additional citation context not shown here]

Horgan, J. R., London, S., Lyu, M. R., Achieving Software Quality with Testing Coverage Measures, IEEE Computer, September, pp. 60-69, 1994.

....the tester has to find the fault while minimizing the cost by minimizing the number of tests. A typical scenario in software testing is that the user tests his program by finding a test bucket (a set of inputs) that meets a certain coverage criteria, e.g. branch coverage or statement coverage [1, 2, 3]. It plausible that in certain situations it might be possible to embed such a set of tests (e.g. the union over all tests buckets that meet branch coverage) in a Poset or in a Tree such that the requirement for covering all tests can be replaced by a requirement for searching in this Poset or ....

Joseph R. Horgan, Saul London, and Michael R. Lyu. Achieving software quality with testing coverage measures. IEEE Transaction on Software Engineering, October 1993.

.... strategies generally suffer from analysis of either small programs or a small 2 number of test subjects (testers) Most suggest that functional testing is a useful preliminary to code testing, creating a base of useful test cases with generally high code coverage [POC93, BS87, Mye78, HH79, KL95, HLL94] It is agreed that tests performed later in the software development process are more expensive to correct [Bei90, Mar91] and hence functional testing coupled with heavy unit testing can be a cheap and effective strategy. Horgan et al. HLL94] also recommend that both functional and structural ....

....high code coverage [POC93, BS87, Mye78, HH79, KL95, HLL94] It is agreed that tests performed later in the software development process are more expensive to correct [Bei90, Mar91] and hence functional testing coupled with heavy unit testing can be a cheap and effective strategy. Horgan et al. HLL94] also recommend that both functional and structural testing be performed but caution that there is a law of diminishing returns whereby the cost of increasing coverage, and hopefully test quality, may not be worthwhile in terms of extra fault detection. 3 Testing Requirements Given the above ....

[Article contains additional citation context not shown here]

J.R. Horgan, S. London, and M.R. Lyu. Achieving Software Quality with Testing Coverage Measures. IEEE Computer, pages 60 -- 69, September 1994.

.... to introduce into the area of software reliability a number of well developed analytical techniques and notions which have been successfully applied to reliability studies in hardware systems[13] Experimental data such as those presented in Table 1 can be easily collected, using tools such as ATAC[4], and utilized to obtain the test coverage function, c(t) by associating coverage information given in column 2 with the cumulative execution time given in column 5. 2.2 Existing Software Reliability Models All the leading software reliability models which are widely used for software quality ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure, " IEEE Computer, vol. 27, no. 9, pp. 60-69, September 1994.

.... to introduce into the area of software reliability a number of welldeveloped analytical techniques and notions which have been successfully applied to reliability studies in hardware systems[14] Experimental data such as those presented in Table 1 can be easily collected, using tools such as ATAC[5], and utilized to obtain the test coverage function, c(t) by associating coverage information given in column 2 with the cumulative execution time given in column 5. 2.1 Existing Software Reliability Models All the leading software reliability models which are widely used for software quality ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure," IEEE Computer, vol. 27, no. 9, pp. 60-69, September 1994.

....test data generation, this suggests that combinatorial optimization may, indeed, be useful for real problems where guesswork does not suffice. 2. Background Empirical results seem to indicate that tests that are selected on the basis of test adequacy criteria are good at uncovering faults (see [Horgan 94] Chilenski 94] DeMillo 94b] for example) Furthermore, test adequacy criteria are objective criteria by which the quality of software tests can be judged. Neither of these benefits can be realized unless adequate test data (i.e. test data that satisfy the adequacy criteria) can be found. ....

Horgan, J., London, S., Lyu, M, "Achieving Software Quality with Testing Coverage Measures, " IEEE Computer Vol. 27, No. 9, pp. 60-69, Sept. 1994.

....the coincidental discovery of tests satisfying new criteria, since they tend to concentrate on restricted areas of the input space. 2 Test adequacy criteria and test data generation Empirical results indicate that tests selected on the basis of test adequacy criteria are good at uncovering faults [Horgan et al. 1994,Chilenski and Miller, 1994,DeMillo and Mathur, 1992] Furthermore, test adequacy criteria are objective measures by which the quality of software tests can be judged. Neither of these benefits can be realized unless adequate test data (i.e. test data that satisfy the adequacy criteria) can be ....

Horgan, J., London, S., and Lyu, M. (1994). Achieving software quality with testing coverage measures. IEEE Computer, 27(9):60--69.

....express unwarranted confidence in the reliability of their programs [6] In spite of this evidence, we find no discussion in the research literature of techniques for testing or assessing the reliability of form based visual programs. In fact, most research on program testing to date (e.g. [11, 13, 17, 25, 33]) is directed at imperative programs. Some recent research [4, 5, 19] considers problems of testing and reliability determination for logic programs written in Prolog. Although the logic paradigm is like the form based paradigm in that both are declarative, several features of the logic paradigm, ....

J. Horgan, S. London, and M. Lyu. Achieving software quality with testing coverage measures. Computer, 27(9):60--9, Sept. 1994.

....satisfying new criteria, since these methods tend to concentrate on restricted areas of the input space. 1. 1 Code coverage as test data generation criteria Empirical results indicate that tests selected on the basis of test adequacy criteria such as code coverage are good at uncovering faults [Horgan et al. 1994,Chilenski and Miller, 1994] Furthermore, test adequacy criteria are objective measures by which the quality of software tests can be judged. Neither of these benefits can be realized unless test data that satisfy the adequacy criteria can be found. Therefore there is a need to generate such ....

Horgan, J., London, S., and Lyu, M. (1994). Achieving software quality with testing coverage measures. IEEE Computer, 27(9):60--69.

....and test coverage include: i) program complexity, ii) software development philosophy or approach, iii) software tools employed, iv) test quality, and (v) test effectiveness. 2. 1 Existing Notions of Coverage The importance of test coverage has been recognized by several researchers [8, 10], and empirical evidence strongly suggests that testing which is carried out without some form of test coverage measurement may fail to sensitize as much as 45 of the code. Software designers and testers must develop effective evaluation tools capable of measuring test coverage and pointing out ....

....Such tools should also provide data which lead to improvements in the quality and effectiveness of a software test as well as information about its adequacy when deciding product release times. There are two types of coverage definitions in literature: control flow and data flow coverage[4, 8, 9, 16]. Each coverage criterion proposed in the literature captures some important aspect of a program s structure. In general, test coverage is a measure of how well a test covers all the potential fault sites in a software product under test. It should be obvious that how one defines a potential ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure," IEEE Computer, pp. 60-69, Sept. 1994.

....in trees is often encountered in program testing or debugging. The bug should be found by searching the program s syntactic tree using a minimal set of queries. As explained in [2] search in trees is also usefull for devising test bucket (a set of inputs that meets a certain coverage criteria [1, 3, 4]) Another area where search in trees appears is searching in classified large treelike data bases (e.g. the Yahoo s data base of the Internet) Finally, search in trees can be used to find a corrupted file in a file system or a data base that has been relocated. Simple checksum like tests can ....

Joseph R. Horgan, Saul London, and Michael R. Lyu. Achieving software quality with testing coverage measures. IEEE Transaction on Software Engineering, October 1993.

....into the program and the tester has to find the fault while using a minimal number of tests. A typical scenario in software testing is that the user tests the program by finding a test bucket (a set of inputs) that meets a certain coverage criteria, e.g. branch coverage or statement coverage [2, 4, 6]. It is plausible that in certain situations it might be possible to embed such a set of tests (e.g. the union over all test buckets that meet branch coverage) in a Poset or in a Tree, such that the requirement for covering all tests can be replaced by a requirement for searching in this Poset or ....

Joseph R. Horgan, Saul London, and Michael R. Lyu. Achieving software quality with testing coverage measures. IEEE Transaction on Software Engineering, October 1993.

....support expressioncount profiling. The major advantages of source level instrumentation are that the instrumented code is portable across compilation environments and that the compiler may be chosen independently of the testing system. A source level instrumentation system for data flow coverage [22] was used in the study presented in Chapter 4. CHAPTER 3. AUTOMATING STRUCTURED TESTING 45 Table 1: Instrumentation library functions Function Arguments Location report activation Module name Module entry report incidence Node index Case labels report expression Decision value, Wrapped around ....

....This chapter uses two metrics for each quality to compare structured testing with all uses data flow coverage and branch coverage. To measure the structured testing and branch coverage criteria, we used the system described in Chapter 3. To measure the all uses criterion, we used ATAC [22] version 3.3.13, treating all uses as the union of block, decision, c uses, and p uses. We performed the comparison using multiple random trials on test programs with known errors, and therefore measured the relative power of the testing criteria themselves rather than the practical effectiveness ....

Horgan, J., London, S., and Lyu, M. Achieving software quality with testing coverage measures. IEEE Computer (Sept. 1994), 60--69.

.... introduce into the area of software reliability a number of welldeveloped analytical techniques and notions which have been successfully applied to reliability studies in hardware systems[1, 6] Experimental data such as those presented in Table 1 can be easily collected, using tools such as ATAC[3], and utilized to obtain the test coverage function, c(t) by associating coverage information given in column 2 with the cumulative execution time given in column 5. 2.2 Existing Software Reliability Models Software failures are known to display the behavior of a Non homogeneous Poisson Process ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure," IEEE Computer, pp. 60-69, Sept. 1994.

.... to introduce into the area of software reliability a number of well developed analytical techniques and notions which have been successfully applied to reliability studies in hardware systems[13] Experimental data such as those presented in Table 1 can be easily collected, using tools such as ATAC[4], and utilized to obtain the test coverage function, c(t) by associating coverage information given in column 2 with the cumulative execution time given in column 5. 2.2 Existing Software Reliability Models All the leading software reliability models which are widely used for software quality ....

J.R. Horgan, S. London, and M.R. Lyu, "Achieving Software Quality with Testing Coverage Measure, " IEEE Computer, vol. 27, no. 9, pp. 60-69, September 1994.

....that, depending on its fitness function, genetic search can be made to concentrate its effort on fruitful areas of the input space. 2 Test adequacy criteria and test data generation Empirical results indicate that tests selected on the basis of test adequacy criteria are good at uncovering faults [Horgan et al. 1994,Chilenski and Miller, 1994,DeMillo and Mathur, 1992] Furthermore, test adequacy criteria are objective measures by which the quality of software tests can be judged. Neither of these benefits can be realized unless adequate test data (i.e. test data that satisfy the adequacy criteria) can be ....

Horgan, J., London, S., and Lyu, M. (1994). Achieving software quality with testing coverage measures. IEEE Computer, 27(9):60--69.

....Introduction Wong et al. 5,6] conduct an experiment in which they seed ten commonly used Unix utilities with faults in order to determine the relative importance of test set size and coverage for assessing fault detection effectiveness, that is, the percentage of faults discovered. Horgan et al. [2] and Piwowarski et al. 4] discuss related issues and suggest that there is a relationship between fault detection effectiveness and coverage measures. They provide data to support their conclusions but do not conduct extensive analyses of their data. We use the data analyzed in [6] to investigate ....

....effectiveness is predicted to within 0.26, with 95 confidence. This interval is quite large, indicating that we are unable to predict fault detection effectiveness precisely in this case. 4.3. Modeling product testing process measures jointly with high levels of coverage Horgan et al. [2] suggest that highly precise predictions should be possible for high levels of coverage, suggesting at least 70 of block coverage as a rule of thumb. In support of this, they present industrial data for which fault detection effectiveness is 100 whenever block coverage is at least 70 . ....

[Article contains additional citation context not shown here]

J. R. Horgan, S. London, and M. R. Lyu, "Achieving Software Quality with Testing Coverage Measures," Computer, September 1994, pp. 60-69.

....Empirical studies. Case studies comparing software development methods are difficult to conduct. Attempts to establish a correlation between the degree of debug testing (usually measured by some structural coverage of unit tests) and the resulting software quality are at best preliminary [7, 13, 22, 9, 17]. On the other side, case studies using formal methods development show great variation, both in the care with which the method is defined and applied and in the results [11] Neither side has any real claim to establishing its case. Analysis of partition testing. A number of theoretical ....

J. Horgan, S. London, and M. Lyu. Achieving software quality with testing coverage. IEEE Computer, 27:60--69, 1994.

.... partition testing techniques to random testing [Duran, Ntafos 1984] Hamlet, Taylor 1990] in analytical treatments [Jeng, Weyuker 1991] Chen, Yu 1994] in analytical comparisons of various testing techniques [Frankl, Weyuker 1993] and in experimental comparisons [Frankl, Weiss 1993] Wong, Horgan et al. 1994]. The expected number of failures during test has also been used as a measure of effectiveness [Frankl, Weyuker 1993] Chen, Yu 1996] Failure finding probability may be a good measure for evaluating test data adequacy criteria (stopping criteria) The best stopping criterion may be the one that ....

....software development methods are difficult to conduct. Attempts to establish a correlation between the degree of debug testing (usually measured by some structural coverage of unit tests) and the resulting reliability in the field are at best preliminary [Dalal, Horgan, Kettenring 1993] [Horgan, London, Lyu 1994], Piwowarski, Ohba et al. 1993] Frankl, Weiss 1993] Lauterbach, Randall 1989] On the other side, case studies using formal methods development show great variation, both in the care with which the method is defined and applied and in the results [Gerhart, Craigen, Ralston 1993] Neither side ....

J. Horgan, S. London, and M. Lyu. Achieving software quality with testing coverage. IEEE Computer, 27:60--69, 1994.

No context found.

J.R. Horgan, S. London, and M.R. Lyu, "Achieving software quality with testing coverage measures," In Computer 1994, pp. 60-69.

....models, however, do not take the coverage of software into consideration. Coverage criteria that capture important aspects of a program s behavior during execution are proposed in [8, 14, 25] and empirical evidence of coverage measures to represent test completeness is illustrated in [13]. Observations, from both empirical and analytical studies, indicate that the reliability predictions made by the SRGMs tend to be too optimistic. 4, 7] Consequently, many researchers postulate that coverage information should be used instead of testing time alone to overcome the difficulty of ....

J. Horgan, S. London, and M. Lyu. Achieving software quality with testing coverage measures. IEEE Computer, 27(9):6--69, September 1994.

No context found.

Horgan, J., London, S., Lyu, M., "Achieving Software Quality with Testing Coverage Measures", IEEE Computer, Vol. 27, No. 9, September 1994, pp. 60-69.

No context found.

Horgan, J.R., London, S., Lyu, M.R.: Achieving software quality with testing coverage measures. IEEE Computer 27 (1994) 60-69

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC