Graphical user interfaces (GUIs), due to their event driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing it is important to “adequately cover ” this interaction space. In this paper, we develop a new family of coverage criteria for GUI testing grounded in combinatorial interaction testing. The key motivation of using combinatorial techniques is that they enable us to incorporate “context” into the criteria in terms of event combinations, sequence length, and by including all possible positions for each event. Our new criteria range in both efficiency (measured by the size of the test suite) and effectiveness (the ability of the test suites to detect faults). In a case study on eight applications, we automatically generate test cases and systematically explore the impact of context, as captured by our new criteria. Our study shows that by increasing the event combinations tested and by controlling the relative positions of events defined by the new criteria, we can detect a large number of faults that were undetectable by earlier techniques.

Abstract — Pair-wise and, more generally, t-wise testing are the most common and powerful combinatorial testing approaches. This paper investigates the effectiveness of the t-wise approach for testing logical expressions in software in terms of its fault-detecting capabilities. Effectiveness is evaluated experimentally using special software tools for generating logical expressions and t-wise test cases, simulating faults in expressions, testing faulty expressions, and evaluating effectiveness of the testing. T-wise testing effectiveness is measured in its totality and for specific types of faults; it is then compared with random testing. A detailed analysis of the experimental results is also provided. Keywords-testing; pair-wise; t-wise; effectiveness; logical expressions; experimental evaluation I.

Most of today’s software users interact with the software through a graphical user interface (GUI), which is a representative of the broader class of event-driven software (EDS). As the correctness of the GUI is necessary to ensure the correctness of the overall software, its quality assurance (QA) is becoming increasingly impor-tant. During software testing, an important QA technique, test cases are created and executed on the software. For GUIs, test cases are modeled as sequences of user input events. Because each possible sequence of user events may potentially be a test case and because today’s GUIs offer enormous flexibility to end users, in principle, GUI testing requires a prohibitively large number of test cases. Any practical test case generation technique must sample the vast GUI input space. Ex-isting techniques are either extremely resource intensive or do not adequately model complex GUI behaviors, thereby limiting fault detection. This research develops new models, algorithms, and metrics for automated GUI test case generation. A novel aspect of this work is its use of software run-time information collected as feedback during GUI test case execution, and used to generate additional test cases that model complex GUI behaviors. One set of empirical studies show that the feedback-directed technique significantly improves upon existing techniques and helps to identify serious problems in fielded GUIs. Another set of studies conducted on in-house software applications show that the test suites generated by the new technique outperform their coverage equivalent counterparts in terms of fault detection. Although the focus of this work is on the GUI domain, the techniques de-veloped are general and are applicable to the broader class of EDS. In fact, this work has already had an impact on research and practice of testing other EDS. In particular, the work has been extended by other researchers to test web applications.

Event-driven software (EDS) is a widely used class of software that takes sequences of events as input, changes state, and outputs new event sequences. Managing the size of tests suites for EDS is difficult as the number of possible event combinations and sequences grow exponentially with the number of events. We propose a new testing technique for EDS that extends software interaction testing. Traditional software interaction testing systematically examines all t-way interactions of parameters for a program. This paper extends this notion to t-way interactions over sequences of events. As a proof-of-concept, we prioritize existing test suites (for four GUIbased programs) by t-way interaction coverage. We compare the rate of fault detection with that of several other prioritization criteria. Our results show that prioritization by interaction coverage has the fastest rate of fault detection in half of our experiments, making the most impact when tests have high interaction coverage.

Event-driven software (EDS) is a widely used class of software that takes sequences of events as input, changes state, and outputs new event sequences. Managing the size of tests suites for EDS is difficult as the number of event combinations and sequences grow exponentially with the number of events. We propose a new testing technique that extends software interaction testing. Traditional software interaction testing systematically examines all t-way interactions of parameters for a program. This paper extends the notion to t-way interactions over sequences of events. The technique applies to many classes of software; we focus on that of EDS. As a proof-of-concept, we prioritize existing test suites for four GUI-based programs by t-way interaction coverage. We compare the rate of fault detection with that of several other prioritization criteria. Results show that prioritization by interaction coverage has the fastest rate of fault detection in half of our experiments, making the most impact when tests have high interaction coverage.

In this paper we describe an approach to use formal analysis tools in conjunction with traditional testing to improve the efficiency of the test generation process. We have developed a technique for the construction of combinatorial test suites, featuring expressive constraints over the models under test and cross coverage evaluation between multiple coverage criteria: combinatorial, structural and fault based. Our approach is tightly integrated with formal logic, since it uses formal logic to specify the system inputs (including the constraints), test predicates to formalize testing as a logic problem, and applies the SAL model checker tool to solve it, and hence to generate combinatorial test suites. Early results of experimental assessment are presented, supported by a prototype tool implementation.

In this paper we describe an approach to use formal analysis tools in conjunction with traditional testing to improve the efficiency of the test generation process. We have developed a technique for the construction of combinatorial test suites, featuring expressive constraints over the models under test and cross coverage evaluation between multiple coverage criteria: combinatorial, structural and fault based. Our approach is tightly integrated with formal logic, since it uses formal logic to specify the system inputs (including the constraints), test predicates to formalize testing as a logic problem, and applies the SAL model checker tool to solve it, and hence to generate combinatorial test suites. Early results of experimental assessment are presented, supported by a prototype tool implementation. 1.