P. G. Frankl, R. G. Hamlet, B. Littlewood, and L. Strigini. Evaluating testing methods by delivered reliability. 24(8):586--601, 1998.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....integration, or system testing. We plan to investigate the derivation of the components of the higher level coverage matrix from coverage matrices at lower levels. Another interesting area of investigation is the effect of test methodology, such as model based testing [1] or partition testing [2, 4], on the test environment parameters . We also intend to perform sensitivity analysis of test cover Replacement Mission Critical Systems Completion m 11 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 Apr 98 Jul 98 Nov 98 Feb 99 May 99 Aug 99 Dec 99 Mar 00 Repalcement Mission ....

P. G. Frankl, R. G. Hamlet, B. Littlewood, and L. Strigini. Evaluating testing methods by delivered reliability. IEEE Trans. on Software Engg., 24(8):586--601, Aug. 1998.

....probable correctness theory [Ham87] calls for uniformly sampling the possible values of all variables. Random testing and operational testing are competitive with or superior to partition testing, debug testing, and other directed testing strategies, at least in terms of delivered reliability [DN84, HT90, FHLS98, FHLS99]. This work makes several reasonable assumptions such as that testers have good but not perfect intuition and that more than a very small number of tests may be performed. Specification coverage is likely to assist in both operational and random testing, permitting improved test coverage, and ....

Phyllis G. Frankl, Richard G. Hamlet, Bev Littlewood, and Lorenzo Strigini. Evaluating testing methods by delivered reliability. 24(8):586--601, August 1998. Special Section: International Conference on Software Engineering (ICSE '97).

....so that the correction effort is cost effective. Several authors have published comparisons between classes of testing methods [1 9] Experimental comparisons have been inconclusive so far. Among comparisons based on modelling, we think the most advanced so far is in [Frankl et al. 1997, Frankl et al. 1998] to which we refer the reader for more extensive background material and whose analysis we seek to extend. Most previous studies of the effectiveness of testing methods used the probability of causing a failure, and thus finding a defect, as a measure of the effectiveness of a test series. This ....

....of a test series. This seems inappropriate when considering testing as a means for improving the software: what really matters is the reliability of the delivered software, hence the improvement that is obtained by applying the given testing method. Frankl and co authors [Frankl et al. 1997, Frankl et al. 1998] (and, previously, Li Malaiya 1994] instead adopted as a measure of test effectiveness the increment in reliability that would be obtained with a given number of tests. Papers [Frankl et al. 1997, Frankl et al. 1998] show that the choice between testing methods depends on rather subtle details ....

[Article contains additional citation context not shown here]

P. Frankl, D. Hamlet, B. Littlewood and L. Strigini, "Evaluating testing methods by delivered reliability", IEEE Transactions on Software Engineering, 24 (8), pp. 586-601, 1998.

....debuggers intuition, if indeed better in finding bugs than mere random choice of test cases, guides them to discover those bugs that contribute heavily to unreliability, so that the correction effort is cost effective. Several authors have published comparisons between classes of testing methods [1 9]. Experimental comparisons have been inconclusive so far. Among comparisons based on modelling, we think the most advanced so far is in [8, 9] to which we refer the reader for more extensive background material and whose analysis we seek to extend. Most previous studies of the effectiveness of ....

....heavily to unreliability, so that the correction effort is cost effective. Several authors have published comparisons between classes of testing methods [1 9] Experimental comparisons have been inconclusive so far. Among comparisons based on modelling, we think the most advanced so far is in [8, 9], to which we refer the reader for more extensive background material and whose analysis we seek to extend. Most previous studies of the effectiveness of testing methods used the probability of causing a failure, and thus finding a defect, as a measure of the effectiveness of a test series. This ....

[Article contains additional citation context not shown here]

P. Frankl, D. Hamlet, B. Littlewood and L. Strigini, "Evaluating testing methods by delivered reliability", IEEE TSE, to appear, 1998.

....one fault associated with each input. One way of thinking of a fault in a program is as the set of all inputs that change from being faulty (i.e. cause a failure when executed) to non faulty (i.e. do not cause a failure when executed) when the program is changed in order to remove a fault [6]. It is now possible to imagine all possible faults that might be in a program. We shall label these with the natural numbers: i: i=1,2,3, Clearly, some of these faults will be more likely to be present in a particular program than others. Let pP i i = fault is present in a randomly ....

....a program s unreliability [7] a fault finding procedure that was very efficient at finding small faults could be very inefficient at finding large ones, and thus at improving reliability. A good fault finding (and removal) procedure is one that tends to improve reliability most efficiently [6]. We now show that similar results to those above also apply to the efficacy of fault finding procedures in improving reliability. Throughout this section we shall assume that when a fault is detected it is removed with certainty. Let i Pi = fault is activated by randomly selected input and ....

[Article contains additional citation context not shown here]

P. Frankl, R. Hamlet, B. Littlewood and L. Strigini, "Evaluating testing methods by delivered reliability", IEEE Trans Software Engineering, 24, pp. 586-601, 1998.

....of separate development means choosing, randomly and independently, possible subsets of this set of possible faults 4 . 3 The concept of a fault corresponding to each failure region is unnecessary for this model, and the term fault itself is loaded with implicit, unrealistic assumptions (cf [8]) but we use it here as a convenient simplification. Thus, the i th fault is present in version A is a convenient short hand for in version A, the ith potential failure region is actually a failure region . 4 It may be useful to recall here that assuming independent choice does not imply ....

P. Frankl, D. Hamlet, B. Littlewood and L. Strigini, "Evaluating testing methods by delivered reliability", IEEE Transactions on Software Engineering, 24, pp. 586-601, 1998.

....not decrease as much as expected and may actually increase. Although the risk reduction resulting from a test set is not directly measurable, under certain assumptions, we can still say something about which testing criteria are better at reducing risk. Frankl, Hamlet, Littlewood and Strigini [4] introduced the notion of a failure region , a subset of the set of failure causing inputs consisting of inputs that are related in the sense that the change made in response to detecting a failure of one of the test cases in the region will fix all of the test cases in the region. This intuition ....

....inputs whose behavior are corrected by the code change to correct the fault together will be thought of as a failure region. It is assumed that if any element in the failure region had been selected as a test case, it would have caused the person to make the same changes to the software. As in [4], we assume that the set of failure causing inputs can be divided into disjoint failure regions, F 1 ; F p , having the property that whenever a test case from F i is executed, the person debugging the program makes a change that exactly removes F i . This implies that this change causes ....

[Article contains additional citation context not shown here]

P.G. Frankl, D. Hamlet, B. Littlewood and L. Strigini. Evaluating testing methods by delivered reliability. IEEE Transactions on Software Engineering, 24(8):586 - 601, Aug, 1998.

....an inspection; accidentally using the wrong variable as target of an assignment statement, during coding; omitting to test under a certain condition that would reveal a fault. The reader will appreciate that for any given failure there may be disagreement in defining the fault that caused it (cf [12]) After agreeing that the fault is, for instance, a missing sanity check on an input value, we may still disagree as to which mistakes caused it any combination of a more explicit specification of required checks, a more defensive programming style, more careful inspections, more thorough ....

P. Frankl, D. Hamlet, B. Littlewood and L. Strigini, "Evaluating testing methods by delivered reliability", IEEE Transactions on Software Engineering, 24, pp. 586-601, 1998.

No context found.

P. G. Frankl, R. G. Hamlet, B. Littlewood, and L. Strigini. Evaluating testing methods by delivered reliability. 24(8):586--601, 1998.

No context found.

P. Frankl, D. Hamlet, B. Littlewood, and L. Strigini. Evaluating Testing Methods by Delivered Reliability. Transactions on Software Engineering, 24(8):586--602, 1998.

No context found.

Phyllis G. Frankl, Richard G. Hamlet, Bev Littlewood, and Lorenzo Strigini. Evaluating testing methods by delivered reliability. IEEE Transactions on Software Engineering, 24(8):586--601, August 1998. Special Section: International Conference on Software Engineering (ICSE '97).

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC