Abstract

Breiman's bagging and Freund and Schapire's boosting are recent methods for improving the predictive power of classifier learning systems. Both form a set of classifiers that are combined by voting, bagging by generating replicated bootstrap samples of the data, and boosting by adjusting the weights of training instances. This paper reports results of applying both techniques to a system that learns decision trees and testing on a representative collection of datasets. While both approaches substantially improve predictive accuracy, boosting shows the greater benefit. On the other hand, boosting also produces severe degradation on some datasets. A small change to the way that boosting combines the votes of learned classifiers reduces this downside and also leads to slightly better results on most of the datasets considered. Introduction Designers of empirical machine learning systems are concerned with such issues as the computational cost of the learning method and the accuracy and ...

Citations

4072	JR: C4.5: Programs for Machine Learning - Quinlan
3144	Classification and Regression Trees - Breiman, Friedman, et al. - 1984
1999	Bagging predictors - Breiman - 1996
1714	Schapire R: A decision-theoretic generalization of online learning and an application to boosting - Freund - 1997
687	Estimation of dependences based on empirical data - Vapnik - 1982
448	Solving multiclass learning problems via errorcorrecting output codes - Dietterich, Bakiri - 1995
173	Classi cation and Regression Trees - Breiman, Friedman, et al. - 1984
112	Learning classification trees - BUNTINE - 1992
97	A Comparative Evaluation of Voting and Meta-Learning of Partitioned Data - K, Stolfo - 1995
80	A system for induction of oblique decision trees - Murty, Kasif, et al. - 1994
59	Addressing the selective superiority problem: Automatic algorithm /model class selection - Brodley - 1993
56	Megainduction: A test flight - Catlett - 1991
52	Lookahead feature construction for learning hard concepts - Ragavan, Rendell - 1993
47	Inductive knowledge acquisition: A case study - Quinlan, Compton, et al. - 1987
44	Automatic parameter selection by minimizing estimated error. In - Kohavi, John - 1995
28	The problem of small disjuncts: its remedy in decision trees - Ting - 1994
28	An Incremental Method for Finding Multivariate Splits for Decision Trees - Utgoff, Brodley - 1990
14	Constructing nominal x-of-n attributes - Zheng - 1995
14	Learning classi cation trees - Buntine
5	Megainduction: a test ight - Catlett - 1991
3	Experiments with a new boosting algorithm. Unpublished manuscript - Freund, Shapire - 1996
3	A comparativeevaluation of voting and meta-learning on partitioned data - Chan, Stolfo - 1995
3	An incremental method for nding multivariate splits for decision trees - E, Brodley - 1990