Comparing the Effectiveness of Testing Methods in Improving Programs: the Effect of Variations in Program Quality
Abstract:
Abstract: We compare the efficacy of different testing methods for improving the reliability of software. Specifically, we use modelling to compare "operational " testing, in which test cases are chosen according to their probability of occurring in actual use of the software, against "debug " testing methods, in which the testers look for test cases which they consider likely to cause failure, or that satisfy some coverage criterion. We base our comparisons on the reliability reached by the program at the end of testing. Differently from previous studies, we consider the probability distribution of the achieved reliability, and thus the probability of satisfying specific requirements, rather than just the average reliability achieved. We take account of two sources of variation: the variation between the actual test histories that are possible for a given program and a given test method; and the fact that different programs start testing with different faults and initial reliability levels. By necessity, we use very simplified models of reality. Yet, we can show some interesting conclusions with important practical consequences. In general, there are stronger arguments in favour of operational testing than previous studies have shown. 1
Citations
| 852 | An introduction to probability theory and its applications – Feller - 1950 |
| 125 | An evaluation of random testing – Duran, Ntafos - 1984 |
| 112 | Partition testing does not inspire confidence – Hamlet, Taylor - 1990 |
| 74 | An experimental comparison of the effectiveness of branch testing and data flow testing – Frankl, Weiss - 1993 |
| 60 | Analyzing partition testing strategy – Weyuker, Jeng - 1991 |
| 54 | A formal analysis of the fault-detecting ability of testing methods – Frankl, Weyuker - 1993 |
| 37 | Evaluating Testing Methods by Delivered Reliability – Frankl, Hamlet, et al. - 1998 |
| 27 | Effect of test set size and block coverage on the fault detection effectiveness – Wong, Horgan, et al. - 1994 |
| 13 | On the expected number of failures detected by subdomain testing and random testing – Chen, Yu - 1996 |
| 8 | On Input Profile Selection for Software Testing – Li, Malaiya - 1994 |
