The testing process is typically systematic in test data selection and test execution. For the most part, however, the effective use of test oracles has been neglected, even though they are a critical component of the testing process. Test oracles prescribe acceptable behavior for test execution. In the absence of judging test results with oracles, testing does not achieve its goal of revealing failures or assuring correct behavior in a practical manner; manual result checking is neither reliable nor cost-effective. We argue that test oracles should be derived from specifications and in conjunction with testing criteria, represented in a common form, and their use made integral to the testing process. For complex, reactive systems, oracles must reflect the multiparadigm nature of the required behavior. Such systems are often specified using multiple languages, each selected for its utility in specifying a particular computational paradigm. Thus, we are developing an approach for derivi...

This paper presents the operational difference technique for generating, augmenting, and minimizing test suites. The technique is analogous to structural code coverage techniques, but it operates in the semantic domain of program properties rather than the syntactic domain of program text. The operational difference technique automatically selects test cases; it assumes only the existence of a source of test cases. The technique dynamically generates operational abstractions (which describe observed behavior and are syntactically identical to formal specifications) from test suite executions. Test suites can be generated by adding cases until the operational abstraction stops changing. The resulting test suites are as small, and detect as many faults, as suites with 100% branch coverage, and are better at detecting certain common faults.

This report presents a new, automatic technique to assess whether replacing a component of a software system by a purportedly compatible component may change the behavior of the system. The technique operates before integrating the new component into the system or running system tests, permitting quicker and cheaper identification of problems. It takes into account the system’s use of the component, because a particular component upgrade may be desirable in one context but undesirable in another. No formal specifications are required, permitting detection of problems due either to errors in the component or to errors in the system. Both external and internal behaviors can be compared, enabling detection of problems that are not immediately reflected in the output. The technique generates an operational abstraction for the old component in the context of the system, and one for the new component in the context of its test suite. An operational abstraction is a set of program properties that generalizes over observed run-time behavior. Modeling a system as divided into modules, and taking into account the control and data flow between the modules, we formulate a logical condition to guarantee that the system’s behavior is preserved across a component replacement. If automated logical comparison indicates that the new component does not make all the guarantees that the old one did, then

The conventional classification of software fault detection techniques as static or dynamic analysis is inadequate as a basis for identifying useful relationships between techniques. A more useful distinction is between techniques that sample the space of possible executions, and techniques that fold the space. The new distinction provides better insight into the ways different techniques can interact, and is a basis for considering hybrid fault detection techniques including combinations of testing and formal verification.

Abstract. Although its importance is widely recognized, testing is seldom done properly. The reasons for this include under-allocation of resources for the testing activity, lack of proper tool support, and developers ’ reluctance towards testing. To tackle these issues, we propose the full automation of the testing process for contract-equipped classes. According to the principles of Design by Contract™, assertions contain the specifications of the software elements. As such, they can be used to ascertain the correctness of these elements. In this paper we discuss the various issues involved in the full automation of the testing process and present our technical solution and its implementation in the tool called AutoTest 1. We also look at ways of improving the current approach. 1

This article focuses on test automation for safety-critical reactive systems. In the first part of the paper we introduce a methodology for specification, design and verification of fault-tolerant systems allowing to combine different methods in a systematic and consistent way, provided that these methods are compositional. The methodology indicates how to "switch" between formal verification and testing during the construction of (possibly large) reactive systems. We introduce the basic notions of testing as far as relevant in the context of reactive systems and relate them to the verification methodology. Part II formally describes our test automation method which is based on Hoare's CSP and takes Hennessy's testing theory as a starting point. It is indicated how this specific method fits into the general approach described in Part I. We introduce CSP test drivers which are trustworthy in the sense that they "approximate" refinement proofs, converging to a full proof with the increas...

As programs evolve, their code increasingly becomes tangled by programmers and requirements. This mosaic quality complicates program comprehension and maintenance. Many of these activities can benefit from viewing the program as a collection of features. We introduce an inexpensive and easily comprehensible summary of program changes called the feature signature and investigate its properties. We find a remarkable similarity in the nature of feature signatures across multiple non-trivial programs, developers and magnitudes of changes. This indicates that feature signatures are a meaningful notion worth studying. We then show numerous applications of feature signatures, establishing their utility. 1

. While Artificial Intelligence techniques have been applied to a variety of software engineering applications, the area of automated software testing remains largely unexplored. Yet, test cases for certain types of systems (e.g., those with command language interfaces and transaction based systems) are similar to plans. We have exploited this similarity by constructing an automated test case generator with an AI planning system at its core. We compared the functionality and output of two systems, one based on Software Engineering techniques and the other on planning, for a real application: the StorageTek robot tape library command language. From this, we showed that AI planning is a viable technique for test case generation and that the two approaches are complementary in their capabilities. Keywords: System testing, AI planning, blackbox testing 1. Automated Test Case Generation Testing consumes a large amount of time and effort in software development. Although critical for ensur...

This paper describes the specification-based testing and analysis tools, and associated processes, that were used to develop and certify safety-critical avionics systems in an industrial organization. These tools comprise an integrated development environment supporting specification acquisition and analysis, requirement-based automatic test vector generation, test coverage analysis, test driver generation, and test results analysis. The paper describes the specification model, method, development environment, and tool qualification approach. The capabilities of the automatic test generator are compared with foundational concepts and related testing strategies and mechanisms. 1.

I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state ” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra [Dijkstra, 1968] 1