Hooker J. N., Needed: An Empirical Science of Algorithms, Operations Research, vol. 42, March-April 1994, 201-212.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....confidence intervals) on input classes that should induce uniform behavior. Ultimately, the role of the skeptic is to be subsumed in a new methodology of experimental design and performance evaluation of combinatorial algorithms. The case for such methods has already been succinctly articulated in [15, 16] as well as demonstrated experimentally in our earlier work [17, 18, 19, 20] We first contrast the proposed approach with the up to date benchmarking experiments on SAT solvers that are being reported on the Web [3] the figure of merit being time to solve on a particular PC. A sample of such a ....

.... SAT solvers has been evaluated experimentally either in terms of randomly generated instances of SAT problems, e.g. 24, 25] or structured instances, such as the instances from the DIMACS set [21] or the SATPLAN set [22] Merits of either approach are subject to on going critique and examination [15, 16, 26, 27, 28]. The traditional way to report results of SAT solvers is the time to solve performance of single instances of a reference formula in conjunctive normal form (cnf) Table 1 represents the traditional organization of such an experimental report. Results of more comprehensive experiments, repeated ....

J.N. Hooker. Needed: An empirical science of algorithms. Operations Research, pages 42(2):201--212, 1994.

....A.M. Segre et al. Artificial Intelligence 85 (1996) 301 319 Since conclusive formal arguments about the performance improvement due to a speedup learning method are difficult to construct, experimental studies provide the only realistic means of detecting or quantifying performance improvement [7]. Data collected in these studies are typically analyzed using some form of hypothesis testing, where the null hypothesis is that there is no difference in performance with or without learning. In [15] we show how several common methodological choices can compromise the reliability of conclusions ....

J.N. Hooker, Needed: an empirical science of algorithms, Oper. Res. 42 (1994) 201-212.

....of algorithms. In a number of computer science fields, we need more and more knowledge about the behavior of the algorithms we design and about the characteristics of benchmarks. This competition is a step in a better and crucial empirical knowledge of SAT algorithms and benchmarks [Hoo94, Hoo96] The aim of this paper is to report what organizers learned during this competition (about solvers, benchmarks and the competition itself) and to publish enough data to allow the reader to make is own opinion about the results. As it was mentioned, in the first half of the last decade, ....

J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--212, 1994.

....confidence intervals) on input classes that should induce uniform behavior. Ultimately, the role of the skeptic is to be subsumed in a new methodology of experimental design and performance evaluation of combinatorial algorithms. The case for such methods has already been succinctly articulated in [15, 16] as well as demonstrated experimentally in our earlier work [17, 18, 19, 20] We first contrast the proposed approach with the up todate benchmarking experiments with SAT solvers that are being reported on the Web [3] the figure of merit being time to solve on a particular PC. A sample of such ....

.... SAT solvers has been evaluated experimentally either in terms of randomly generated instances of SAT problems, e.g. 24, 25] or structured instances, such as the instances from the DIMACS set [21] or the SATPLAN set [22] Merits of either approach are subject to on going critique and examination [15], 16] 26] 27] 28] The traditional way to report results of SAT solvers is the time to solve performance of single instances of a reference formula in conjunctive normal form (cnf) Table 1 represents the traditional organization of such an experimental report. Results of more ....

J.N. Hooker. Needed: An empirical science of algorithms. Operations Research, pages 42(2):201--212, 1994.

....has recently gained much attention. New workshops (ALENEX, WAE) and journals (ACM J. of Experimental Algorithmics) have been installed and established conferences (e.g. SODA, ESA) explicitly call for experimental work. Using the scientific method as a basis for algorithmics was proposed by Hooker [6]. McGeoch, Precup and Cohen [12] give heuristic algorithms for finding upper bounds on measured function values which are found to be reliable within a factor n. They stress that finding more accurate bounds would be futile in general. This is no contradiction to the examples given in Sect. 4 ....

J. Hooker. Needed : An empirical science of algorithms. Operations Res., 42(2):201-- 212, 1994.

....as I elaborate them. Indeed, although there is much common agreement on what makes good experimental analysis of algorithms, certain aspects have been the subject of debate, such as the relevance of running time comparisons. For other points of view, see such recent papers as [BGK 95, CS00, Hoo93, McG96, MM99, RU01] This particular paper was constructed using my earlier short draft [Joh96] as a template and adding in material I have developed in the process of giving talks expanding on that draft. The points of view expressed in this paper are my own, although they have been refined (and ....

J. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--212, March-April 1993.

.... with mathematical software [14] Despite these guidelines being established over twenty years ago, recent articles have suggested that they have had little or no impact; and that the field is still failing to produce reports, which even conform to minimal standards of reproducibility reporting [15,16]. So what is the basic problem Although it is difficult to be certain, the most likely candidate is time. It is expensive to reproduce all the information required to allow such replications. Additionally, software engineering, in general, faces even greater problems in capturing all the ....

J N Hooker, "Needed: An empirical science of algorithms", Operations Research, Vol. 42, No. 2, pp. 201-212, 1994

.... either in terms of randomly generated instances of SAT problems, e.g. 28, 11] or structured instances, such as the instances from the DIMACS set [30] or the SATPLAN set [23] Merits of either approach are subject to on going critique and examination [10] 24] 26] in particular, and [21], 22] 25] in general. The papers [21] and [22] succinctly articulate the case for careful experimental design an approach adopted for the experiments with SAT problems in this paper. The experimental design methodology presumes availability of well defined classes of experimental subjects. ....

....instances of SAT problems, e.g. 28, 11] or structured instances, such as the instances from the DIMACS set [30] or the SATPLAN set [23] Merits of either approach are subject to on going critique and examination [10] 24] 26] in particular, and [21] 22] 25] in general. The papers [21] and [22] succinctly articulate the case for careful experimental design an approach adopted for the experiments with SAT problems in this paper. The experimental design methodology presumes availability of well defined classes of experimental subjects. In our earlier work we identified and ....

J. Hooker, Needed: An empirical science of algorithms, Operations Research, (1994), pp. 42(2):201--212.

....hard, and therefore provide a fair basis for comparison of different algorithms. They can also provide a basis for furthering our understanding of the way algorithms behave on both random and real problems. We hope therefore to contribute to the developing empirical science of algorithms [10]. Phase transitions have received considerable attention in the AI community [2, 13, 7] Whilst random problems are typically easy to solve, hard random problems can be found at a phase transition [2] AI researchers now routinely use such problems to benchmark satisfiability and constraint ....

J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42:201--212, 1994.

....search steps, i.e. we get run length distributions (RLDs) instead of run time distributions. Note, that obtaining run length distributions for single instances does not involve significantly higher computation times than to get a stable estimate for the mean performance of an algorithm. Hooker [12,13] criticises that the empirical analysis of algorithms usually remains at the stage of simply collecting data and argues that, analogous to empirical methodology used in other sciences, one should furthermore attempt to formulate hypotheses based on this data which, in turn, can be experimentally ....

J.N. Hooker. Needed: An Empirical Science of Algorithms. Operations Research, 42(2):201--212, 1994.

....5) Thus, bad results should also be reported in any exhaustive and objective account. All experimental sciences are based on bad results as well as good results and it is clear that bad results can be really interesting if they help to understand the behavior of the algorithm being presented [16] . Another point related to objectivity is the choice of the result parameters used for the comparison of algorithms. It is seldom the case that a single parameter is sufficient to obtain a fair comparison. For instance, computing the only sum of CPU times on a set of benchmark instances doesn t ....

J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42:201--212, 1994.

....Correlation is indeed important, but there are other distributional factors that can influence performance of solution methods as well. For example, other factors that matter include distribution family (Loulou and Michaelides 1979) and the range of coefficient values (Yang 1994; Reilly 1998a) Hooker (1994) advocates the development on an empirical science of algorithms. By recognizing the characteristics of synthetic optimization problems and understanding how those characteristics affect the performance of solution methods, one can better interpret the results of computational experiments and ....

Hooker, J.N. 1994. Needed: an empirical science of algorithms. Operations Research 42 (2): 201-212.

....have polynomial algorithms. It is of importance to basic AI research to know why some distributions in the average, typical or worst case are intractable whereas others are not. This will help us to identify the core of the search problem which algorithms face and thereby improve them. Hooker [10] suggested that one advantage of an empirical science of algorithms is that it can investigate how performance is related to the characteristics of the problem. In this paper we take an empirical approach to study the emergence of structural characteristics which may be 1 Division of Informatics, ....

J. N. Hooker, `Needed: An empirical science of algorithms', Operations Research, 42(2), 201--212, (1994).

....such as when applying WalkSAT and using an optimal setting of wp to hard Random 3 SAT problem instances, the RTDs are well approximated by exponential distributions , which can be tested for statistical significance. Therefore, this type of RTD based analysis follows suggestions made by Hooker [30, 31], that the empirical analysis of algorithms should not remain at the stage of simply collecting data. Rather, analogous to empirical methodologies used in other sciences, one should attempt to formulate hypotheses based on this data which, in turn, can be experimentally refuted or validated. As we ....

J.N. Hooker. Needed: An Empirical Science of Algorithms. Operations Research, 42(2):201-- 212, 1994.

....has recently gained much attention. New workshops (ALENEX, WAE) and journals (ACM J. of Experimental Algorithmics) have been installed and established conferences (e.g. SODA, ESA) explicitly call for experimental work. Using the scientific method as a basis for algorithmics was proposed by Hooker [6]. McGeoch, Precup and Cohen [12] give heuristic algorithms for finding upper bounds on measured function values which are found to be reliable within a factor p n. They stress that finding more accurate bounds would be futile in general. This is no contradiction to the examples given in Sect. 4 ....

J. Hooker. Needed : An empirical science of algorithms. Operations Res., 42(2):201-- 212, 1994.

....give a good picture of the algorithm s behavior. Besides the average and standard deviation, we have also reported the minimum, median, and maximum. Hooker has advocated an empirical approach to algorithm analysis that involves hypothesis formation and prediction, in analogy with natural sciences [Hoo94]. Many heuristics are proposed in the literature, with informal explanations of why they work well. Hooker believes that the explanations should be examined more critically, and proposes a framework for doing so: if the explanation is correct , then we should be able to make new predictions and ....

J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--12, March-April 1994.

....as this can easily lead to heavily biased evaluations and assessments of the usefulness of specific algorithms. Hence, SATLIB s benchmark collection comprises mostly instances which are known to be hard for a wide range of SAT algorithms. To avoid some well known pitfalls of benchmarking [10, 11], benchmark libraries should generally not be static, but allow to be updated with new challenging problems. As elaborated in Section 4, SATLIB therefore encourages the submission (and retraction) of benchmark problems and thus lives of the contribution of people involved in SAT research. In the ....

J.N. Hooker. Needed: An Empirical Science of Algorithms. Operations Research, 42(2):201-- 212, 1994.

....among programming language families (logic, functional, imperative, object oriented, rule based, constraint based) Another important example is algorithms theory. The present theory has many drawbacks; in particular, it does not account for the behavior of algorithms on typical problems[7]. A more accurate theory that applies to modern computers would be valuable. A prerequisite for competition among theories is falsifiability. Unfortunately, computer science theorists rarely produce falsifiable theories; they tend to pursue mathematical theories that are disconnected from the real ....

John N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--212, March 1994.

....such as satisfiability and constraint satisfaction. Where we have not followed these maxims, we have suffered as a result. 1 Introduction The empirical study of algorithms is a relatively immature field with many technical and scientific problems. We support the calls of McGeoch (1986,1996) Hooker (1994), and others for a more scientific approach to the empirical study of algorithms. Our contribution in this paper is colloquial. We admit to a large number of mistakes in conducting our research. While painful, we hope that this will encourage others to avoid these mistakes, and thereby to develop ....

J. N. Hooker. 1994. Needed: An empirical science of algorithms. Operations Research 42:201--212.

....Analysis environment We considered using mathematical analysis or measurements of a real implementation as the evaluation method, but simulation turned out to be the best choice. Mathematical average case analysis using realistic input models is beyond the limits of the current analytical methods [20, 32] for algorithms like disk tc. On the other hand, simulation takes orders of magnitude less time than measuring a real system that does actual disk I O; thus we can study a large number of inputs. We have implemented an environment, called DBE, for the performance analysis of memory I O intensive ....

J.N. Hooker. Needed: an empirical science of algorithms. Operations Research, 42(2):201-- 212, March-April 1994.

.... [14] Jackson et al.[29] Lee et al. [34] and Hoaglin and Andrews[26] Even so, Hauck and Anderson[24] found little or no improvement in the reporting of simulation experiments by statisticians nearly a decade after the recommendations of Hoaglin and Andrews were published and, more recently, Hooker[27] has commented that Not even the minimal standards of reproducibility are observed by most authors. Ironically, bearing in mind the work of Shneiderman et al. which was dismissive of the utility of flowcharts, Lee at al[34] suggest use of flowcharts as one way of describing the detail of ....

....activities to understand the role of such pragmatic maintenance and to determine if it is a good or a bad thing. There is no getting away from explaining the human component in software engineering experiments that have human subjects. We regard this paper as being complementary to Hooker s[27] paper on the need for an empirical science of algorithms and the paper by Basili et al.[5] on experimentation in software engineering through our extension to their framework. We recommend that educational programmes place greater emphasis on laboratory work with students designing, executing, ....

J N Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--212, 1994.

....goal of scheduling algorithms. Introduction With the recent burgeoning of interest in empirical approaches to artificial intelligence [AAAI Empirical Workshop, 1994; ECAI Empirical Workshop, 1996] a number of authors have made calls for a more scientifically rigorous basis for such research [Hooker, 1994; Cohen, 1995] While much of the field of the empirical study of algorithms is relatively immature, empirical scheduling research may be especially so due to the widespread (and currently unmet) demand for scheduling solutions to real problems in industry and elsewhere. This demand has ....

....reproduction of results, and hypotheses testing to develop a deeper understanding of the behaviour of scheduling techniques across a wide body of problems. In short, we urge that the calls for a rigorous empirical science of algorithms be heeded particularly well in the scheduling community [Hooker, 1994]. In this paper we present a number of pitfalls that we have observed and experienced in our research. The data used to illustrate each pitfall comes from experiments we have performed over the past few years. The motivation for writing this paper is similar to that of earlier work describing ....

Hooker, J. N. (1994). Needed: An empirical science of algorithms. Operations Research, 42:201-- 212.

....will come a time when no one takes us seriously . In later articles, Denning cites the field of performance evaluation as a positive example of experimental CS research [4, 5] Several articles describe the role of experimental research in branches of CS, e.g. machine learning [11] algorithms [8], or software engineering [7] The latter article is quite critical of software engineering research and states: There are far too few examples of moderately effective research. Baldwin and Koomen [2] discuss practicing experimental computer science during CS education. In 1990, Iyer [10] ....

....experimental computer science during CS education. In 1990, Iyer [10] writes that experimental CS is a relatively new, yet fast developing area and finds: It is indeed encouraging to see that there is substantial research going on in this important area. Four years later, however, Hooker [8] notes that experimental research is dramatically underdeveloped in algorithms research. He states that most experimental efforts fall short of science on several levels and continues It is symptomatic of the situation in OR and computer science one cannot publish reports that an algorithm does ....

John N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--212, March 1994.

....effectively repair a theory which we showed to be inaccurate to a slight degree. However this repair is empirical. If our results could be used to help develop a more refined and accurate theory, it would be a pleasing validation of the empirical science of algorithms, as called for by Hooker [14]. ....

J.N. Hooker, Needed: An empirical science of algorithms, Operations Research 42 (2) (1994) 201-212.

....effectively repair a theory which we showed to be inaccurate to a slight degree. However this repair is empirical. If our results could be used to help develop a more refined and accurate theory, it would be a pleasing validation of the empirical science of algorithms, as called for by Hooker [14]. ....

J.N. Hooker, Needed: An empirical science of algorithms, Operations Research 42 (2) (1994) 201-212.

....are usually relatively simple (like algorithm A outperforms algorithm B ) and the analytical methodology used is both easy to apply and powerful enough to get the desired results. Or is it really Recently, there has been some severe criticism regarding the empirical testing of algorithms (Hooker, 1994; Hooker, 1996; McGeoch, 1996) It has been pointed out that the empirical methodology that is used to evaluate and compare algorithmsdoes not reflect the standards which have been established in other empirical sciences. Also, it was argued that the empirical analysis of algorithms should not ....

Hooker, J. (1994). Needed: An Empirical Science of Algorithms.

....search algorithms, particularly for SAT and CSPs, have given rise to a large number of empirical studies in the past few years. At the same time there has been some severe criticism of the methodology that has been applied in empirical computer science in general, and in empirical AI in particular (Hooker 1996; 1994). Since the theoretical understanding of the behavior of stochastic local search algorithms is still very limited, empirical methods are the most important means of studying these algorithms. Due to their significance in designing, evaluating, and understanding stochastic local search ....

Hooker, J. 1994. Needed: An Empirical Science of Algorithms. Operations Research 42(2):201--212.

....(i) collect a set of benchmark problems that are representative of problems that arise in practice, and (ii) randomly generate problems and investigate how algorithmic performance depends on problem characteristics . and learn to predict how an algorithm will perform on a given problem class (Hooker, 1994). For IA networks, there is no existing collection of large benchmark problems that actually arise in practice as opposed to, for example, planning in a toy domain such as the blocks world. As a start to a collection, we propose an IA network with 145 intervals that arose from a problem in ....

Hooker, J. N. (1994). Needed: An empirical science of algorithms. Operations Research, 42, 201--212.

....output metrics. We used sequential procedures [15] that automatically yield the desired statistical con dence. Thus, our experimental approach is related to the recent interest in developing an empirical science of algorithms that is based on rigorous methods of statistics and experimental design [16]. Ganguly et al. 17] used a similar approach in their study of transitive closure computation. In the experiments, we compared the sizes of the interval and chain representations and two list representations. We also studied the complexity of constructing the interval representation when ....

....have to choose probabilities that are not realistic. Second, given an algorithm and an input model, computing the average case execution time or space is often not possible. As a general rule, average case analysis using realistic input models is beyond the limits of the current analytical methods [16, 35]. The average case analysis may be easier for probabilistic algorithms, which use the distribution of the input model in their behavior. If the inputs are simple combinatorial structures like binary trees, a divide and conquer strategy may be possible. In analyzing the average case performance of ....

[Article contains additional citation context not shown here]

J.N. Hooker. Needed: an empirical science of algorithms. Operations Research, 42(2):201 212, March-April 1994.

....We applied exactly the same branch and bound algorithm to the original problem (without decomposition) and compared the computation time to that of the Benders algorithm. This provided a controlled experiment in which the e ect of decomposition could be isolated, a general approach advocated in [24, 26]. Because the master problem does not have the traditional inequality constraints, we solved it with modi ed branch and bound algorithm. We branched on y j s with fractional values as well as on the possible values of the right hand side of each Benders cut (39) More precisely, for k = 1; ....

Hooker, J. N., Needed: An empirical science of algorithms, Operations Research 42 (1994) 201-212.

....the insight, as in done routinely in other empirical sciences, and deduce from it precise consequences that can be put to the test. I will give this sort of experimentation the deliberately honorific name scientific testing to distinguish it from competitive testing. I discuss elsewhere [9] how empirical models might be constructed and defend them as a viable and necessary alternative to a purely deductive science of algorithms. My main object in this paper is to show that scientific testing can avoid or substantially alleviate many of the evils that now stem from competitive ....

Hooker, J. N., Needed: An empirical science of algorithms, Operations Research 42 (1994) 201-212.

.... branching, such as the Horn relaxation algorithm, the branch and cut algorithm of Hooker and Fedjki [15] and the hypergraph algorithms of Gallo and Pretolani [10] A second purpose of this study is to demonstrate some elements of the empirical paradigm for the study of algorithms recommended in [14]. Rather than simply compare branching rules in computational tests, we formulate models that purport to explain the behavior of branching rules. We view these models as empirical theories analogous to those developed in physics or chemistry. They do not and are not intended to lead to ....

Hooker, J. N., Needed: An empirical science of algorithms, Operations Research 42 (1994) 201-212.

No context found.

Hooker J. N., Needed: An Empirical Science of Algorithms, Operations Research, vol. 42, March-April 1994, 201-212.

No context found.

J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201-212, 1994.

No context found.

J. Hooker. Needed: An empirical science of algorithms. Operations Research, 42:201--212, 1994.

No context found.

J.N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42:201--212, 1994.

No context found.

J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201--212, 1994.

No context found.

J. N. Hooker. Needed: an empirical science of algorithms. Operations Research, 42(2):201--212, 1994.

No context found.

J.N. Hooker, Needed: An empirical science of algorithms, Operations Research 42 (2) (1994) 201-212.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC