Although detractors of functional programming sometimes claim that functional programming is too difficult or counterintuitive for most programmers to understand and use, evidence to the contrary can be found by looking at the popularity of spreadsheets. The spreadsheet paradigm, a first-order subset of the functional programming paradigm, has found wide acceptance among both programmers and end users. Still, there are many limitations with most spreadsheet systems.

This article presents a testing methodology that adapts data flow adequacy criteria and coverage monitoring to the task of testing spreadsheets. To accommodate the evaluation model used with spreadsheets, and the interactive process by which they are created, our methodology is incremental. To accommodate the users of spreadsheet languages, we provide an interface to our methodology that does not require an understanding of testing theory. We have implemented our testing methodology in the context of the Forms/3 visual spreadsheet language. We report on the methodology, its time and space costs, and the mapping from the testing strategy to the user interface. In an empirical study, we found that test suites created according to our methodology detected, on average, 81% of the faults in a set of faulty spreadsheets, significantly outperforming randomly generated test suites

Testing is an important process that is performed to support quality assurance. Testing activities support quality assurance by gathering information about the nature of the software being studied. These activities consist of designing test cases, executing the software with those test cases, and examining the results produced by those executions. Studies indicate that more than fifty percent of the cost of software development is devoted to testing, with the percentage for testing critical software being even higher. As software becomes more pervasive and is used more often to perform critical tasks, it will be required to be of higher quality. Unless we can find efficient ways to perform effective testing, the percentage of development costs devoted to testing will increase significantly. This report briefly assesses the state of the art in software testing, outlines some future directions in software testing, and gives some pointers to software testing resources.

Is it possible to achieve some of the benefits of formal testing within the informal programming conventions of the spreadsheet paradigm? We have been working on an approach that attempts to do so via the development of a testing methodology for this paradigm. Our "What You See Is What You Test" (WYSIWYT) methodology supplements the convention by which spreadsheets provide automatic immediate visual feedback about values by providing automatic immediate visual feedback about "testedness". In previous work we described this methodology; in this paper, we present empirical data about the methodology's effectiveness. Our results show that the use of the methodology was associated with significant improvement in testing effectiveness and efficiency, even with no training on the theory of testing or test adequacy that the model implements. These results may be due at least in part to the fact that use of the methodology was associated with a significant reduction in overconfidence. Keywor...

End-user programming has become the most common form of programming today. However, despite this growth, there has been little investigation into bringing the benefits of software visualization to end-user programmers. Evidence from the spreadsheet paradigm, probably the most widely used end-user environment, reveals that end users' programs often contain faults. We would like to integrate software visualization into these end-user environments to help end users deal with the reliability issues in their programs. Towards this end, we have devised several fault localization visualization techniques for spreadsheets. This paper describes these techniques and reports the results of a formative study--using tests created by end users--to investigate how these fault localization techniques compare. Our results reveal some strengths and weaknesses of each technique, and provide insights into the cost-effectiveness of each technique for the interactive world of end-user spreadsheet development.

Despite their ability to help with program correctness, assertions have been notoriously unpopular---even with professional programmers. End-user programmers seem even less likely to appreciate the value of assertions; yet end-user programs suffer from serious correctness problems that assertions could help detect. This leads to the following question: can end users be enticed to enter assertions? To investigate this question, we have devised a curiosity-centered approach to eliciting assertions from end users, built on a surprise-explain-reward strategy. Our follow-up work with end-user participants shows that the approach is effective in encouraging end users to enter assertions that help them find errors.

Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein.

Where the development, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support rigorous controlled experimentation with testing techniques, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of testing techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with software testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have on controlled experimentation with testing techniques.

Although there has been recent research into ways to design environments that enable end users to create their own programs, little attention has been given to helping these end users systematically test their programs. To help address this need in spreadsheet systems---the most widely used type of end-user programming language---we previously introduced a visual approach to systematically testing individual cells in spreadsheet systems. However, the previous approach did not scale well in the presence of largely homogeneous grids, which introduce problems somewhat analogous to the array-testing problems of imperative programs. In this paper, we present two approaches to spreadsheet testing that explicitly support such grids. We present the algorithms, time complexities, and performance data comparing the two approaches. This is part of our continuing work to bring to end users at least some of the benefits of formalized notions of testing, without requiring knowledge of testing beyond a naive level.

Although there has been considerable research into ways to design visual programming environments to improve the processes of creating new programs and of understanding existing ones, little attention has been given to helping users of these environments test their programs. This feature would be particularly important for systems aimed at end users, since testing is the primary device they use to determine whether their programs are correct. To help address this need, we introduce two visual approaches to testing large grids in spreadsheet systems. This work scales up a visual testing methodology we previously developed for individual cells. The approaches are tightly integrated into Forms/3, a visual spreadsheet language, and communication with the user happens solely through the use of checkbox devices and coloring mechanisms. The intent of this work is to bring to end users at least some of the benefits of formalized notions of testing, without requiring knowledge of testing beyond a naive level. 1.