S.L. Salzberg, On comparing classifiers: a critique of current research and methods, Data Mining and Knowledge Discovery 1 (1999) 1-12.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....on the binomial distribution (such as McNemar s test) 33] At first examination, the binomial distribution is also particularly attractive because it is a simple, easily understood, and wellstudied distribution. Its use for evaluation has been discussed for classifiers [1] 10] 15] 41] [42] and for evaluation and tuning in a variety of other domains, including inductive learning systems [18] and speech recognition algorithms [19] However, there can be direct and measurable consequences when the underlying data departs from srs. In [26] Kish discusses some quantitative ....

Steven L. Salzberg. On comparing classifiers: A critique of current research and methods. Data Mining and Knowledge Discovery, 1:1-- 12, 1999. 12

....presented can be applied. In most studies in the machine learning community, the goal is to compare different algorithms. The criterion in those cases is often how well those algorithms perform on standard data sets (e.g. UC Irvine repository [20] or in a particular domain. As pointed out in [28], such comparisons of algorithms may be misleading since the performance of the classifiers they produce depends strongly on the specific domain in which they are applied (e.g. features, training, etc. For this reason, many researchers in the Machine Learning encourage the empirical evaluation ....

....[0, 1, 2, 4, 11, 12] CV: 0.43 T: 22.75 R: 116 [1, 2, 4, 7, 12, 15, 17, 37] CV: 0.43 T: 23.63 R: 118 [1, 2, 7, 12, 20, 37] CV: 0.58 T: 23.14 R: 115 1, 2, 7, 9, 11, 12, 19, 20, 37] CV: 1.30 T: 22.36 R: 115 [1, 2, 3, 6, 12, 18, 21, 30, 37] CV: 1. 29 T: 17.60 R: 112 [1, 2, 10, 11, 15, 16, 17, 19, 20, 22, 23, 24, 25, 26, 27, 28, 30, 35, 38] CV: 0.45 T: 27.03 R: 95 Top Grass [0, 2, 7, 8, 9, 21] CV: 0.14 T: 14.86 R: 142 [2, 5, 8, 9, 10, 11, 19, 35, 38] CV: 0.43 T: 16.25 R: 145 [0, 2, 7, 8, 9, 21,22] CV: 0.14 T: 15.06 R: 145 [1, 2, 6, 8, 11, 12, 38] CV: 1.01 T: 16.23 R: 128 [0, 2, 3, 4, 7, 9, 10, ....

S. L. Salzberg, "On Comparing Classifiers: A Critique of Current Research and Methods", Data Mining and Knowledge Discovery, 1, pp. 1-12, 1999.

....4.12 9.08 Table 3: The results of cross validation are shown in units of percent accuracy, including the mean and sample standard deviation. HN is the probabilistic model of Horton Nakai. All trials of kNN are for k = 7. cross validation overlap heavily and thus the trials are not independent (Salzberg 1995). Despite these observations, the t test has been shown empirically to discriminate adequately (Dietterich 1996) Results A summary of the accuracies of the different classifiers is given in table 3 for E.coli and table 4 for yeast. Accuracies for the smaller E.coli dataset were estimated with ....

Salzberg, S. 1995. On comparing classifiers: A critique of current research and methods.

....method as better when it narrowly won. Consider n non ties, with s of those n wins were method A beating method B, so that method A seems to be better. If there is actually no differences between the two methods, then p = q = 0:5 in Equation 5, and the probability that A and B are equally good is [23]: Classes 0 Cutoff 10 Cutoff 25 Cutoff 50 Cutoff 100 Cutoff 2 0.9449 0.9505 0.9521 0.9528 0.9513 3 0.8458 0.8648 0.8663 0.8640 0.8514 4 0.7370 0.7529 0.7454 0.7390 0.7163 5 0.6266 0.6541 0.6450 0.6285 0.5909 6 0.5349 0.5468 0.5292 0.5140 0.4757 7 0.4955 0.4930 0.4776 0.4549 0.4117 ....

Steven L. Salzberg. On comparing classifiers: A critique of current research and methods. Technical Report CS-1995-06, John Hopkins University, May 1995.

....and run both GA speciation methods on them. The best cutoff values are asterisked. Using their best cutoff values, Table 6 shows which method classified better. Are the differences in Table 6 significant We use a binomial test (as each set of letters are drawn randomly from the alphabet) [19], requiring the number of times one method is better, and ignoring ties. We try two different definitions of a tie in Table 6. The first: a method is better even if it classifies only one extra letter correctly. As this can be a narrow margin, we next take a tie as a difference in classification ....

S. L. Salzberg. On comparing classifiers: A critique of current research and methods. Technical Report CS-1995-06, John Hopkins University, 1995.

No context found.

S.L. Salzberg, On comparing classifiers: a critique of current research and methods, Data Mining and Knowledge Discovery 1 (1999) 1-12.

No context found.

Steven Salzberg. On comparing classifiers: A critique of current research and methods. Kluwer Academic Publishers, Boston, 1999.

No context found.

S.L. Salzberg, "On Comparing Classifiers: A Critique of Current Research and Methods," Data Mining and Knowledge Discovery, 1:1-12, 1999.

No context found.

Salzberg, S. L. 1998. 'On comparing classifiers: A critique of current research and methods', Data Mining and Knowledge Discovery vol. 1, p.1 -- 12.

No context found.

Salzberg, S., On Comparing Classifiers: A critique of current research and Methods. Technical Report JHU-95/06, Department of Computer Science, Johns Hopkins University, May 1995.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC