Download:
|
by Foster Provost, Tom Fawcett, Ron Kohavi
In Proceedings of the Fifteenth International Conference on Machine Learning
http://www.hpl.hp.com/personal/Tom_Fawcett/papers/ICML98-final.ps.gz
Add To MetaCart
Abstract:
We analyze critically the use of classification accuracy to compare classifiers on natural data sets, providing a thorough investigation using ROC analysis, standard machine learning algorithms, and standard benchmark data sets. The results raise serious concerns about the use of accuracy for comparing classifiers and draw into question the conclusions that can be drawn from such studies. In the course of the presentation, we describe and demonstrate what we believe to be the proper use of ROC analysis for comparative studies in machine learning research. We argue that this methodology is preferable both for making practical choices and for drawing scientific conclusions. 1
Citations
|
3307
|
C4.5: Programs for machine learning
– Quinlan
- 1993
|
|
2524
|
Classification and regression trees
– Breiman, Friedman, et al.
- 1984
|
|
2195
|
UCI Repository of Machine Learning Databases
– Blake, Merz
- 1998
|
|
1504
|
Bagging Predictors
– Breiman
- 1996
|
|
343
|
A study of cross-validation and bootstrap for accuracy estimation and model selection
– Kohavi
- 1995
|
|
321
|
Approximate Statistical Test for Comparing Supervised Classification Learning Algorithms
– Dietterich
- 1998
|
|
308
|
Supervised and unsupervised discretization of continuous features
– Dougherty, Kohavi, et al.
- 1995
|
|
248
|
Beyond independence: conditions for the optimality of the simple bayesian classier
– Domingo, Pazzani
- 1996
|
|
211
|
The Use of the Area Under the ROC Curve in the Evaluation of Machine Learning Algorithms
– Bradley
- 1997
|
|
191
|
Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions
– Provost, Fawcett
- 1997
|
|
155
|
Classi cation and Regression Trees
– Breiman, Friedman, et al.
- 1984
|
|
132
|
A conservation law for generalization performance
– Schaffer
- 1994
|
|
131
|
Data mining using MLC++: a machine learning library in
– Kohavi, Sommerfield, et al.
- 1997
|
|
130
|
Signal Detection Theory and ROC Analysis
– Egan
- 1975
|
|
128
|
Measuring the Accuracy of Diagnostic Systems
– Swets
- 1988
|
|
96
|
On comparing classifiers: pitfalls to avoid and a recommended approach. Data Mining. and Knowledge Discovery 1:317–327
– Salzberg
- 1997
|
|
70
|
Evaluation of diagnostic systems: methods from signal detection theory
– Swets, Pickett
- 1982
|
|
46
|
Pruning decision trees with misclassification costs
– Bradford, Kunz, et al.
- 1998
|
|
44
|
Robust classification systems for imprecise environments
– Provost, Fawcett
- 1998
|
|
41
|
Adaptive fraud detection. Data Mining and Knowledge Discovery
– Fawcett, Provost
- 1997
|
|
12
|
A conservation law for generalization performance
– er, C
- 1994
|
|
10
|
Analysis and visualization of classi er performance: Comparison under imprecise class and cost distributions
– Provost, Fawcett
- 1997
|
|
7
|
On Comparing Classi ers: Pitfalls to Avoid and a Recommended Approach
– Salzberg
- 1997
|
|
4
|
Tailoring rulesets to misclassificatioin costs
– Catlett
- 1995
|
|
3
|
Cost sensitive learning bibliography. Available: http://ai.iit.nrc.ca/ bibliographies/cost-sensitive.html
– Turney
- 1996
|
|
3
|
The use of ROC curves in test performance evaluation. Arch Pathol Lab
– Beck, Schultz
- 1986
|
|
2
|
Pruning decision trees with misclassi cation costs
– Bradford, Kunz, et al.
- 1998
|
|
2
|
Robust classi cation systems for imprecise environments
– Provost, Fawcett
- 1998
|
|
1
|
Tailoring rulesets to misclassi catioin costs
– Catlett
- 1995
|
|
1
|
Egan.(1975) Signal Detection Theory and ROC Analysis. Series in Cognitition and Perception
– P
|
