Abstract

We review accuracy estimation methods and compare the two most common methods: cross-validation and bootstrap. Recent experimental results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifiers (model selection), ten-fold cross-validation may be better than the more expensive leaveone-out cross-validation. We report on a largescale experiment -- over half a million runs of C4.5 and a Naive-Bayes algorithm -- to estimate the effects of different parameters on these algorithms on real-world datasets. For cross-validation, we vary the number of folds and whether the folds are stratified or not; for bootstrap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets similar to ours, the best method to use for model selection is ten-fold stratified cross validation, even if computation power allows using more folds.

Citations

3143	Classification and Regression Trees - Breiman, Friedman, et al. - 1983
1783	An Introduction to the Bootstrap - Tibshirani, R - 1993
662	Repository of machine learning databases - Murphy, Aha - 1994
655	C4.5: Programs for - Quinlan - 1993
463	Stacked generalization - Wolpert - 1992
285	An analysis of Bayesian classifiers - Langley, Iba, et al. - 1992
173	Classi cation and Regression Trees - Breiman, Friedman, et al. - 1984
148	Estimating the error rate of a prediction rule: improvement on cross-validation - Efron - 1983
140	A Conservation Law for Generalization Performance - Schaffer - 1994
91	MLC++: A machine learning library in C - Kohavi, John, et al. - 1994
45	Submodel selection and evaluation in regression. the x-random case - Breiman, L, et al. - 1992
37	The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework - Wolpert - 1994
18	Estimating the accuracy of learned concepts - Bailey, C - 1993
16	Off-training set error and a priori distinctions between learning algorithms - Wolpert - 1995
15	An analysis of bayesian classi ers - Langley, Iba, et al. - 1992
14	Decision tree pruning : Biased or optimal - Weiss, Indurkhya - 1994
13	A conservation law for generalization performance - er, C - 1994
6	Small sample error rate estimation for k-nearest neighbor classi ers - Weiss - 1991
3	Linear model selection via crossvalidation - Shao - 1993
2	Bootstrap techniques for error estimation", IEEE transactions on pattern analysis and machine intelligence PAMI-9(5 - Jain, Dubes, et al. - 1987
1	Bootstrap techniques for error estimation", IEEE transactions on pattern analysis and machine intelligence PAMI-9(5 - Jain, Dubes, et al. - 1987