The dearth of published empirical data on major industrial systems has been one of the reasons that software engineering has failed to establish a proper scientific basis. In this paper we hope to provide a small contribution to the body of empirical knowledge. We describe a number of results from a quantitative study of faults and failures in two releases of a major commercial system. We tested a range of basic software engineering hypotheses relating to: the Pareto principle of distribution of faults and failures; the use of early fault data to predict later fault and failure data; metrics for fault prediction; and benchmarking fault data. For example, we found strong evidence that a small number of modules contain most of the faults discovered in pre-release testing, and that a very small number of modules contain most of the faults discovered in operation. However, in neither case is this explained by the size or complexity of the modules. We found no evidence to support previous claims relating module size to fault density, nor did we find evidence that popular complexity metrics are good predictors of either fault-prone or failure-prone modules. We confirmed that the number of faults discovered in pre-release testing is an order of magnitude greater than the number discovered in 12 months of operational use. We also discovered fairly

The failure of safety-critical systems can result in catastrophic loss of life and property. Hence, it is necessary to assure the reliability of these systems to a high degree of confidence before they are put into operational use. However, at these extreme levels of ultra-high reliability requirements, typically failures rates of less than 10 \Gamma7 failures per hour, errors in the specification and in estimates of the operational profile become significant factors. An approach that has been suggested in practice is to use secondary and tertiary software that meet ultra-high reliability requirements but at a reduced functionality as compared with the primary software. Two major problems are (a) how to select appropriate functionality for the non-primary versions and (b) how to determine when to invoke these backup versions. In this paper, we present a unified approach for handling these two problems. It starts with a rigorous method for assessing ultra-high reliability requirements...

This paper addresses the problem of measuring the reliability of safety-critical software. One theoretically sound approach is the statistical sampling method which, however, has some practical drawbacks. The two most serious objections are the large number of test cases needed to attain a reasonable confidence in the reliability estimate and the sensitivity of the reliability estimate to errors in assessing the operational profile. One way of dealing with both of these issues is to use formal methods. The most obvious method is to verify complete program paths. This is especially effective if high usage paths are verified. However, the verification of complete paths is viable only when there is a high confidence in the correctness of the specification. In this paper, we develop a method of integrating sourcecode transformationwith statistical sampling to get a practical way of measuring software reliability. Several transformation steps were identified, including data structure transf...

The statistical sampling method is a theoretically sound approach for measuring the reliability of safety-critical software, such as control systems for nuclear power plants, aircrafts, space vehicles, etc. It has, however, some practical drawbacks, two of which are the large number of test cases needed to attain a reasonable confidence in the reliability estimate and the sensitivity of the reliability estimate to variations in the operational profile. One way of dealing with both of these issues is to combine statistical sampling with formal methods and attempt to verify complete program paths. This combination becomes especially effective if high usage paths are verified. However, the verification of complete paths is difficult to perform in practice and viable only when there is a high confidence in the correctness of the specification. In this paper we identify program transformations and partial proofs which have a measurable impact on the reliability assessment procedure. These m...

Stochastic Petri nets (SPNs) have emerged over the years as a favored approach for performance and dependability modeling and evaluation. Their usual utilization assumes that systems specification and design do not evolve, in opposition to real-life. This paper is aimed at a preliminary exploration of how to take advantage of the existing body of results on SPNs for modeling the evolution of computer systems, i.e. to model non-stationary stochastic processes. It focuses on dependability evolutions which result from successive releases.