I. P. Gent, S. A. Grant, E. MacIntyre, P. Prosser, P. Shaw, B. M. Smith, and T. Walsh. How not to do it. Technical Report Report 97.27, 1997.  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... generators, most often the standard C library s drand48( There are several important papers on random number generators that should be read by everyone who does numerical simulations [133, 134, 135, 136] Other important papers to read for anyone considering doing numerical simulations are [137, 138, 139] 4.5 Monte Carlo simulation of polymers As an example of computer simulations of a system that is not a spin model, this section will present a study of the number of occurrences of small sub structures in polymers. Lattice models of polymers have attracted much attention from physicists. Part ....

I. P. Gent, S. A. Grant, E. MacIntyre, P. Prosser, P. Shaw, B. M. Smith and T. Walsh, How Not To Do It. APES eprint.

....some guidelines for experimental comparisons of search algorithms, we refrain from stating general principles for scientific experiments such as reproducibility. Of course, other researcher have made similar and related observations, and the work of others has also influenced our guidelines (e.g. [2]) So they are, of course, not all new. A major motivation for compiling these guidelines is to make it easier for someone having developed a new search algorithm to compare it with existing algorithms on a fair basis. In addition, we argue for reflecting on the experimental results. 2.1 ....

....is to be preferred. Of course, this requires that implementations are provided by their authors (preferably on a non commercial basis) 1 Once you have published a useful search algorithm, make it available to others as well 2 Develop a fast implementation of your new algorithm. In [2] there is the recommendation to make the code fast enough to allow making lots of experiments on large problems. We agree to this recommendation, and according to our experience the implementation of the algorithms compared may need to be fast even in order to avoid misleading results. 1 At this ....

[Article contains additional citation context not shown here]

I. Gent, S. Grant, E. MacIntyre, P. Prosser, P. Shaw, B. Smith, and T. Walsh. How not to do it. Research Report 97.27, University of Leeds, Leeds, UK, 1997.

....processes and problem specific content. It is the role of software engineering to approach these generic process deficiencies. The adaptation of processes and the incorporation of problem specific information will be addressed later. Gent and Walsh evaluate neural network experimental practice [8] and identify several generic weaknesses in current neural network experimental evaluation. 1. a lack of software configuration management; 2. the absence of extensive independent testing in a planned objective manner; 3. infeasibility of implementation; 4. failure to collect metrics regarding ....

....of implementation; 4. failure to collect metrics regarding implementation; 5. lack of repeatability of the process. As pointed out by Gent and Walsh, this list of lessons is far from comprehensive. It was not intended to be. There are still many mistakes for us to make in the future. [8] Adya and Collopy [1] identify three criteria that should be considered when assessing any technique as it applies to any system that behaves as a predictor. The criticisms can be levelled at neural networks: 1. Forecasts from a proposed model should perform at least as well as some ....

I. Gent and T. Walsh. How not to do it. Technical report, AAAI Workshop on Experimental Evaluation of Reasoning and Search Methods, July 1994. 9

....but also using a competent implementation. In our experience, proposed improvements often look good if applied to weak or slow implementations, but may actually worsen strong or fast implementations (cf. e.g. the maxims Do make it fast enough and Do measure CPU time given by Gent et al. [19]) An incorrectly implemented testbed can lead to incorrect conclusions, which then waste the research community s efforts to reproduce the improvement that purportedly results from the new technique. In the following, we illustrate the danger of irreproducible and possibly meaningless ....

....or understood partitioning testbench. Our intest is to raise awareness of a very much underestimated root cause of a poor testbench, namely, the failure to understand implicit implementation decisions that can dominate quality runtime tradeoffs (cf. Do report important implementation details [19]) A corollary is that researchers must take pains to identify and clearly describe such implicit decisions in order for results to be reproducible. With respect to the Fiduccia Mattheyses algorithm, we believe that many implementation pitfalls correspond to implicit decisions underspecified ....

[Article contains additional citation context not shown here]

I. P. Gent, S. A. Grant, E. MacIntyre, P. Prosser, P. Shaw, B. M. Smith and T. Walsh, "How Not To Do It", research report 97-27, Univ. of Leeds School of Computer Studies, May 1997.

....for experimental comparisons of search algorithms. We refrain, however, from stating general principals for scientific experiments such as reproducibility. Of course, other researcher have made similar and related observations, and the work of others has also influenced our guidelines (e.g. [7]) So they are, of course, not all new. A major motivation for compiling these guidelines is to make it easier for someone having developed a new search algorithm to compare it with existing algorithms on a fair basis. Once search algorithms are compared in a more standardized way, it will be ....

....is to be preferred. Of course, this requires that implementations are provided by their authors (preferably on a non commercial basis) 5 Once you have published a useful search algorithm, make it available to others as well 6 ffl Develop a fast implementation of your new algorithm. In [7] there is the recommendation to make the code fast enough to allow making 5 At this point, there is again a good opportunity to thank Rich Korf and Anup Sen for providing us with their code. The strategy of JAIR (Journal of Artificial Intelligence Research) to let implemented code also be added ....

[Article contains additional citation context not shown here]

I.P Gent, S.A. Grant, E. MacIntyre, P. Prosser, P. Shaw, B.M. Smith, and T. Walsh. How not to do it. Research Report 97.27, University of Leeds, Leeds, UK, 1997.

....we have very rarely succeeded in following this advice, and have learnt from it only to the extent of giving ourselves deadlines where none is imposed from outside. Do check your references This advice is easily given but harder to follow. For example, in an earlier version of this very paper [Gent and Walsh 1994c] we gave the incorrect page numbers for a classic in our field. This was because, to our shame, we had cribbed the reference from another paper rather than the source itself. 7 Conclusions We have presented some of the lessons we have learnt in studying NPcomplete problems experimentally. With ....

I. Gent and T. Walsh. 1994c. How not to do it. Research Paper 714, Dept. of Artificial Intelligence, Edinburgh.

No context found.

I. P. Gent, S. A. Grant, E. MacIntyre, P. Prosser, P. Shaw, B. M. Smith, and T. Walsh. How not to do it. Technical Report Report 97.27, 1997.

No context found.

I.P. Gent and T. Walsh. How Not To Do It. APES Technical Report. 1995. http://www.dcs.st-and.ac.uk/#apes/1995.html

No context found.

I. P. Gent, S. A. Grant, E. MacIntyre, P. Prosser, P. Shaw, B. M. Smith and T. Walsh, "How Not To Do It", research report 97-27, Univ. of Leeds School of Computer Studies, May 1997.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC