Abstract

Abstract. Many KDD systems incorporate an implicit or explicit preference for simpler models, but this use of “Occam’s razor ” has been strongly criticized by several authors (e.g., Schaffer, 1993; Webb, 1996). This controversy arises partly because Occam’s razor has been interpreted in two quite different ways. The first interpretation (simplicity is a goal in itself) is essentially correct, but is at heart a preference for more comprehensible models. The second interpretation (simplicity leads to greater accuracy) is much more problematic. A critical review of the theoretical arguments for and against it shows that it is unfounded as a universal principle, and demonstrably false. A review of empirical evidence shows that it also fails as a practical heuristic. This article argues that its continued use in KDD risks causing significant opportunities to be missed, and should therefore be restricted to the comparatively few applications where it is appropriate. The article proposes and reviews the use of domain constraints as an alternative for avoiding overfitting, and examines possible methods for handling the accuracy–comprehensibility trade-off.

Citations

6694	Statistical Learning Theory - Vapnik - 1998
6514	Elements of Information Theory - Cover, Thomas - 1991
3909	Neural Networks for Pattern Recognition - Bishop - 1995
1998	Bagging predictors - Breiman - 1996
1581	Estimating the dimension of a model - SCHWARZ - 1978
1368	Text categorization with support vector machines - Joachims - 1998
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
891	Modeling by shortest data description - Rissanen - 1978
800	Fast effective rule induction - Cohen - 1995
787	Field Theory - Wit, Smith - 1986
752	Learning Bayesian networks: the combination of knowledge and statistical data - Heckerman, Geiger, et al. - 1995
606	W: Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. The Annals of Statistics - Schapire, Freund, et al. - 1998
537	Neural networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992
486	Pazzani M: On the optimality of the simple Bayesian classifier under zero-one loss - Domingos - 1997
463	Stacked generalization - Wolpert - 1992
417	Bayesian interpolation - MacKay - 1992
385	Very simple classification rules perform well on most commonly used datasets - Holte - 1993
274	Inferring decision trees using the minimum description length principle. Information and Computation 80.227–248 - QUINLAN, RIVEST - 1989
270	An information measure for classification - Wallace, Boulton - 1968
251	Bagging, boosting, and C4.5 - Quinlan - 1996
221	Structural risk minimization over data-dependent hierarchies - Shawe-Taylor, Bartlett, et al. - 1940
207	Quantifying inductive bias: AI learning algorithms and Valiant’s learning framework - Haussler - 1988
202	Mining association rules with item constraints - Srikant, Vu, et al. - 1997
167	The Need for Biases in Learning Generalizations - Mitchell - 1990
166	On bias, variance, 0/1 - loss, and the curse-of-dimensionality, Data Mining and Knowledge Discovery 1 - Friedman - 1997
156	An Empirical Comparison of Pruning Methods for Decision–Tree Induction - Mingers - 1989
154	Extracting support data for a given task - Scholkopf, Burges, et al. - 1995
140	A Conservation Law for Generalization Performance - Schaffer - 1994
134	A new SQL-like operator for mining association rules - Meo, Psaila, et al. - 1996
131	Error-correcting output coding corrects bias and variance - Kong, Dietterich - 1995
116	Overfitting avoidance as bias - Schaffer - 1993
110	Theory refinement combining analytical and empirical methods. Artificial Intelligence 66:311--344 - Ourston, Mooney - 1994
90	Boosting in the limit: Maximizing the margin of learned ensembles - Grove, Schuurmans - 1998
87	A belief-driven method for discovering unexpected patterns - Padmanabhan, Tuzhilin - 1998
84	Creating advice-taking reinforcement learners - Maclin, Shavlik - 1996
82	Learning from Hints in Neural Networks - Abu–Mostafa - 1990
80	An empirical evaluation of bagging and boosting - Maclin, Opitz - 1997
79	Using general impressions to analyze discovered classification rules - Liu, Hsu, et al. - 1997
77	Unifying instance–based and rule–based induction - Domingos - 1996
76	Cameron-Jones. Oversearching and Layered Search in Empirical Learning - Quinlan, M - 1995
67	Multiple Comparisons in Induction Algorithms," Machine Learning (to appear). Excellent discussion of bias inherent in selecting an input. Explore http://www.cs.umass.edu/~jensen/papers. How to cite B.Rajasekhar - Jensen, Cohen - 1999
65	Extracting Comprehensible Models from Trained Neural Networks - Craven - 1996
64	Metarule-guided mining of multi-dimensional association rules using data cubes - Kamber, Han, et al. - 1997
62	Randomization Tests - Edgington
61	Grammatically biased learning: Learning logic programs using an explicit antecedent description language - Cohen - 1994
53	Datamine: Application Programming Interface and Query Language for Database Mining - Imielinski, Virmani, et al. - 1996
52	MineSet: An integrated system for data mining - Brunk, Kelly, et al. - 1997
52	Exploring the decision forest: An empirical investigation of Occam's razor in decision tree induction - Murphy, Paxzani - 1994
51	Knowledge acquisition from examples via multiple models - Domingos - 1997
51	Further experimental evidence against the utility of Occam’s Razor - Webb