Kearns, M., & Mansour, Y. (1996) On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1), 1999, pp 109128. Also in Proceedings ACM Symposium on the Theory of Computing, 1996, ACM Press, pp.459-468.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....(eds. Machine Learning: ECML 2001 Proc. 12th European Conference, ECML 01 (Freiburg, Germany) The weak learning model or boosting theory [16, 6] has been able to ooeer an analytical explanation for the practical success of top down induction of decision trees (subsequently DTs for short) [9, 12]. Earlier attempts to explain the success of DT learning in theoretical models have not been successful. Even though the weak learning framework may better suit analyzing and designing practical learning algorithms than the PAC model and its variants, one must exercise care in drawing conclusions ....

....obtained by using BPs rather than DTs is that their training error is guaranteed to decline exponentially in the square root of the size of the program. When DT learning algorithms are viewed as boosting algorithms, the training error declination is only polynomial in the size of the tree [9, 12]. The learning algorithm for BPs is basically very similar to the algorithms used in top down induction of DTs [3, 15] It greedily searches for good splits of the nodes in the last level of the evolving program. The central dioeerence between a BP and a DT is that in the former branches may grow ....

[Article contains additional citation context not shown here]

Kearns, M., Mansour, Y.: On the boosting ability of top-down decision tree learning algorithms. J. Comput. Syst. Sci. 58 (1999) 109128

.... in the well known CART TM package [24] It is worthwhile to remark that Gini s criterion, as well as more recent criteria such as Schapire Singer s Z criterion [25] have been rigorously proven to be very ecient measures to grow decision trees, in particular more accurate than the accuracy itself [26]. Furthermore, in our case, a convenient statistical test, which we now describe, allows to estimate with con dence whether a feature subset can be preferred to another one in our algorithm. 3.5 The Statistical Test The previous criterion U tot allows to estimate the information level of the ....

....gives now a greater threshold probability p t 0:11. 18 The preceding experimental results show that the accuracy is not an accurate criterion to be optimized, since it is outperformed by the RCG. Such results were previously observed and theoretically explained in decision tree induction. In [26], a formal proof is given which explains why the Gini criterion and the entropy should be optimized instead of the accuracy when a top down induction algorithm is used to grow a decision tree. Their theoretical results support the claim according to which maximizing the accuracy should be done ....

[Article contains additional citation context not shown here]

M.J. Kearns, Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. Proceedings of the 28 Annual ACM Symposium on the Theory of Computing (1996) pp. 459-468. 21

....that decision tree like boosting algorithms can boost accuracy arbitrarily close to the noise rate. In particular, we analyze a modi ed version of the branching programs booster of Mansour and McAllester [12] which built on a boosting analysis of decision trees due to Kearns and Mansour [9]. We refer to the boosting algorithm from [12] as the MM boosting algorithm, and to our modi ed version as the MMM boosting algorithm. We next show that in general it is not possible to black box boost accuracy past the noise rate. The results described above assume that the boosting algorithm ....

.... In this section we describe a particular boosting algorithm and analyze its performance in the absence of noise (i.e. when = 0) The algorithm we describe here is essentially the branching program booster of Mansour and McAllester [12] which built on ideas from Kearns and Mansour s paper [9]) and we henceforth refer to it as the MM boosting algorithm. Our goal here is to set the stage for our analysis of the MMM algorithm in the presence of noise, which we give in Sections 4 and 6. Our presentation and analysis of the MM algorithm are slightly di erent from that of [12] in order to ....

[Article contains additional citation context not shown here]

M. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and System Sciences, 58(1):109-128, 1999.

....states on an arbitrary domain that some algorithm satis es the weak learning hypothesis better than another one. So far, only an induction scheme has seemingly brought an experimental accurate answer to the building of weak hypotheses, and has been supported by theoretical comparison studies [KM96]. This scheme has previously been successful to build formulas such as decision trees [Qui94] decision lists [NJ98] and, of course, our simple rules in our experiments with SFboost and AdaBoost. Figure 2 presents the margin distribution for SFboost over one run, for three problems of the UCI ....

M.J. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. Proceedings of the 28 Annual ACM Symposium on the Theory of Computing, pages 459-468, 1996.

....of # t ) and chooses the split with smallest value. The decision node corresponding to this split of this prediction node is then added to the tree. The distribution over training examples is then updated to emphasize examples which the newly added decision node classi es incorrectly. In [42], Kearns and Mansour analyze decision tree learning algorithms in terms of boosting. Their analysis suggests an algorithm similar to the one presented here. 4.3 Interpreting alternating decision trees In this section we demonstrate how alternating decision trees can be interpreted using the same ....

....new insights. Theoretically, the analysis seems weaker than it could be. The representation of decision trees as thresholded convex combinations of leaf functions is convenient, but somewhat arti cial. An alternative approach to understanding decision trees in terms of boosting is presented in [23, 42]. Alternating decision trees There is much work to be done before alternating decision trees can be considered a practical alternative to existing methods. Firstly, the method of construction is currently very computationally intensive. Friedman et al. 37] describe a technique called weight ....

M. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the 28th ACM Symposium on the Theory of Computing, pages 459-468. ACM Press, 1996.

....earlier observations on the way in which boosting analyses bear practical signi cance. 1 Introduction The weak learning model or boosting theory [16, 6] has been able to o er an analytical explanation for the practical success of top down induction of decision trees (subsequently DTs for short) [9, 12]. Earlier attempts to explain the success of DT learning in theoretical models have not been successful. Even though the weak learning framework may better suit analyzing and designing practical learning algorithms than the PAC model and its variants, one must exercise care in drawing conclusions ....

....obtained by using BPs rather than DTs is that their training error is guaranteed to decline exponentially in the square root of the size of the program. When DT learning algorithms are viewed as boosting algorithms, the training error declination is only polynomial in the size of the tree [9, 12]. The learning algorithm for BPs is basically very similar to the algorithms used in top down induction of DTs [3, 15] It greedily searches for good splits of the nodes in the last level of the evolving program. The central di erence between a BP and a DT is that in the former branches may grow ....

[Article contains additional citation context not shown here]

Kearns, M., Mansour, Y.: On the boosting ability of top-down decision tree learning algorithms. J. Comput. Syst. Sci. 58 (1999) 109128

....impurity or heterogeneity of each subset of training examples. All such functions are qualitatively similar, with a unique maximum at p = 0:5, and equal minima at p = 0 and p = 1. Drummond and Holte [2000] have shown that for twovalued attributes the impurity function 2 p p(1 p) suggested by Kearns and Mansour [1996] is invariant to changes in the proportion of different classes in the training data. We prove here a more general result that applies to all discretevalued attributes and that shows that related impurity functions, including the Gini index [Breiman et al. 1984] are not invariant to base rate ....

M. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the Annual ACM Symposium on the Theory of Computing, pages 459--468. ACM Press, 1996.

....k is straightforward. However, with more complex classi ers this is not always the case. Some possibilities for C(v; k) include the Gini index and the entropy (e.g. Venables and Ripley 1997) but in a boosting context the criterion preferred by Schapire and Singer (1998) is the one introduced by Kearns and Mansour (1996), which analytically minimises the bound on the training error. This is given by C(v; k) 2 N X j=1 q W j W j ; 1) where N = 2 when using decision stumps, W j = P i D t (i)I [signfh(x i )g = 1 and x i 2 X j ] the weighted fraction of points classi ed as 1 in partition j, ....

Kearns, M. and Mansour, Y. (1996). On the boosting ability of top-down decision tree learning algorithms, Proc. 28th Annual ACM Symp. Theory Comp.

....of that feature, and the hypothesis generated when a sample reaches that node. A terminal node is specified by a hypothesis only. In CART and C4.5, the splitting criterion is based on an information theoretic value [3, 12] Let us look at another splitting criterion first discussed by others [4, 8, 14]. We have looked at this criterion (termed the Z criterion) over a range of difficult and hard problems and found it superior to that of C4.5. Let us examine the denominator of equation (1) of Figure 3. Its value is 1 ( exp( m ttit i ZDiyhi = 2) It is important because one can ....

Michael Kearns, and Yishay Mansour, On the Boosting Ability of Top-Down Decision Tree Learning Algorithms, Proceedings of the Twenty-Eight Annual Symposium on the Theory of Computers, 1996

....with low training error as has usually been done in the past, we show that, theoretically, our methods work best when combined with a weak learner which minimizes an alternative measure of badness. For growing decision trees, this measure turns out to be identical to one earlier proposed by Kearns and Mansour (1996). Although we primarily focus on minimizing training error, we also outline methods that can be used to analyze generalization error as well. Next, we show how to extend the methods described above for binary classification problems to the multiclass case, and, more generally, to the multi label ....

....in growing a decision tree, rather than the Gini index or an entropic function. In other words, the decision tree could be built by greedily choosing the split which causes the greatest drop in the value of the function given in Eq. 10) In fact, exactly this splitting criterion was proposed by Kearns and Mansour (1996). Furthermore, if one wants to boost more than one decision tree then each tree can be built using the splitting criterion given by Eq. 10) while the predictions at the leaves of the boosted trees are given by Eq. 9) 4.2. Smoothing the predictions The scheme presented above requires that we ....

Kearns, M., & Mansour, Y. (1996). On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing.

....6 OPEN QUESTIONS Our results give evidence of the broad utility of boosting algorithms such as Adaboost. A natural question is how much further this utility extends: are there simple boosting based PAC versions of other standard learning algorithms We note in this context that Kearns and Mansour [23] have shown that various heuristic algorithms for top down decision tree induction can be viewed as instantiations of boosting. Another goal is to construct more powerful boosting based PAC algorithms for linear threshold functions. All of the algorithms discussed in this paper have an inverse ....

M. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms, in "Proc. 28th Symp. on Theor. of Comp.," (1996), 459-468.

....Z D i y h i t t i t i m = exp( 1 (1) and is crucial to defining a new splitting criterion for trees because we can show [20] that minimizing Z minimizes the training error. Thus we shall attempt to minimize Z for each tree. This function has been investigated elsewhere [13] but did not show the improvement we show here. We attribute this to the different pruning procedures described later. Growing Trees Using the Z Function Trees are constructed using a training set and pruned using a pruning set. Trees consist of a series of nodes each with connections to two ....

Michael Kearns and Yishay Mansour, "On the boosting ability of top-down decision trees learning algorithms", Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing, Morgan Kaufmann, 1996.

....and k is straightforward. However, with more complex classifiers this is not always the case. Some possibilities for C(v; k) include the Gini index and the entropy (e.g. Venables and Ripley 1997) but in a boosting context the criterion preferred (Schapire and Singer 1998) is the one introduced by Kearns and Mansour (1996) and given by C(v; k) 2 N X j=1 q W j W j Gamma ; 1) where N = 2 when using decision stumps, W j Sigma = P i D t (i)I [signfh(x i )g = Sigma1 and x i 2 X j ] the weighted fraction of points classified as Sigma1 in partition j, where the dependence of W j Sigma on t; v ....

Kearns, M. and Mansour, Y. (1996). On the boosting ability of top-down decision tree learning algorithms, Proc. 28th Annual ACM Symp. Theory Comp.

....uses a small sample at the root node and it enlarges it progressively as the tree grows. Here, we propose an alternative way to select the appropriate sample size needed at every node by using an instantiation of our algorithm that we describe in the following. We will follow the notation from [6]. For simplicity, we assume that we want to construct a decision tree that approximates and unknown function f : X f0; 1g using a training data set S X. Furthermore, we assume that the class of node functions F is xed a priori and it is nite and small, for instance just the input variables ....

....whole function U . If G(q) is the Gini index, then its derivative is G 0 (q) 4(1 2q) and for any 0 q 1, jG 0 (q)j 4. Thus, by the Mean Value Theorem, the Lipschitz constant is 4. If G(q) is the binary entropy H(q) or the improved splitting criterion G(q) 2 q q(1 q) presented in [6], its derivatives are not bounded in the [0; 1] range and therefore it cannot be a xed constant that works for all the possible values. However, suppose that we ignore the input values very close to 0 or 1 and we consider, for instance, the interval [0:05; 0:95] then both functions have Lipschitz ....

[Article contains additional citation context not shown here]

Michael Kearns and Yishay Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proc. of 28th STOC, pp. 459-468, 1996.

....As to overcome those problems Lopez de Mantaras [12] has proposed to use another, but closely related evaluation function, and Quinlan [17] has recently been compelled to change the evaluation of numerical attributes in C4.5. Still, analysis of this function has been surprisingly scarce (c.f. [5, 10, 12]) In the following we take few steps towards understanding where the pitfalls lie in using this function. Firstly, let us review how gain ratio is actually used in C4.5. In the first phase, the gain ratio maximizing partition is computed for each attribute. Then, instead of picking the gain ratio ....

M. Kearns and Y. Mansour, On the boosting ability of top-down decision tree learning algorithms. In: Proc. Twenty-Eighth Annual ACM Symposium on Theory of Computing. ACM Press, New York, NY, 1996, 459--468.

....attributes. Association rules that classify a target attribute will provide us with valuable information which may in turn helps understand relationships between conditional and target attributes. Association rules are used to derive good decision trees. As pointed out by Kearns and Mansour [14], the recent popularity of decision trees such as C4.5 by Quinlan [22] is due to their simplicity and e#ciency and one of the advantage of using decision trees is potential interpretability to humans. One dimensional association rules for categorical attributes can be e#ciently obtained [20] On ....

M. Kearns and Y. Mansour, On the boosting ability of top-down decision tree learning algorithms, Journal of Computer and System Sciences, 58 (1999) 109-128.

....data and refine the solution. It s very difficult to estimate how fast the algorithm will give a satisfactory solution. ffl The complexity of the model. The best known theoretical upper bounds on sample size suggest that the training set size may need to be immense to assure good accuracy [12, 19]. 2. In many real applications, customers insist that all data, not just a sample of the data, must be processed. Since the data are usually obtained from valuable resources at considerable expense, they should be used as a whole throughout the analysis. Therefore, designing a scalable classifier ....

M. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the 28th ACM Symposium on the Theory of Computing, pages 459--468, 1996.

....We cover very little of this work in the survey, primarily due to the author s ignorance. Proceedings of the annual COLT conferences and International Conferences on Machine Learning (ICML) are good starting points to explore this work. A few good papers, to get a flavor for this work, are [169, 285, 177, 178, 148]. Work on learning Bayesian or inference networks from data is closely related to automatic decision tree construction. There are an increasing number of papers on the former topic, although the similarities with tree induction are usually not pointed out. For a good discussion of decision tree ....

Michael Kearns and Yishay Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing, pages 459--468, Philadelphia, Pennsylvania, 1996. 40 SREERAMA K. MURTHY

....with low training error as has usually been done in the past, we show that, theoretically, our methods work best when combined with a weak learner which minimizes an alternative measure of badness. For growing decision trees, this measure turns out to be identical to one earlier proposed by Kearns and Mansour (1996). Although we primarily focus on minimizing training error, we also outline methods that can be used to analyze generalization error as well. Next, we show how to extend the methods described above for binary classification problems to the multiclass case, and, more generally, to the multi label ....

....in growing a decision tree, rather than the Gini index or an entropic function. In other words, the decision tree could be built by greedily choosing the split which causes the greatest drop in the value of the function given in Eq. 10) In fact, exactly this splitting criterion was proposed by Kearns and Mansour (1996). Furthermore, if one wants to boost more than one decision tree then each tree can be built using the splitting criterion given by Eq. 10) while the predictions at the leaves of the boosted trees are given by Eq. 9) 4.2. Smoothing the predictions The scheme presented above requires that we ....

Kearns, M., & Mansour, Y. (1996). On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing.

....tree learning algorithms are popular in practice it seems hard to quantify their success in a theoretical model. It is fairly easy to see that even if the target function can be described using a small decision tree, such an algorithm may fail to find a good approximation. Kearns and Mansour [6] used the weak learning hypothesis to show that such algorithms perform boosting. This provides a theoretical justification for decision tree learning similar to justifications that have been given for various other boosting algorithms, such as AdaBoost [4] Most decision tree learning algorithms ....

....be the fraction of the sample reaching leaf , i.e. jS j=jSj. We define q to be the fraction of the pairs hx; f(x)i in S for which f(x) 1. The training error of T , denoted ffl(T ) is P 2L(T ) p min(q ; 1 Gamma q ) 3 The Weak Learning Hypothesis and Boosting Here, as in [6], we view top down decision tree learning as a form of Boosting [8, 3] Boosting describes a general class of iterative algorithms based on a weak learning hypothesis. The classical weak learning hypothesis applies to classes of Boolean functions. Let H 2 be the subset of branching functions h 2 H ....

[Article contains additional citation context not shown here]

Michaeil Kearns and Yishai Mansour. On the boosting ability of top-down decision tree learning. In Proceedings of the Twenty-Eighth ACM Symposium on the Theory of Computing, pages 459--468, 1996.

....justification for them, and suggest specific modifications that yield fast, practical and principled methods for pruning with proven error guarantees. Combined with earlier results proving non trivial performance guarantees for the common greedy, top down growth heuristics in the model of boosting [5], it is fair to say that there is now a solid theoretical basis for both the top down and bottom up passes of many standard decision tree learning algorithms. 2 Framework and Preliminaries We consider decision trees over an input domain X . Each such tree has binary tests at each internal node, ....

Michael Kearns and Yishay Mansour. On the Boosting Ability of Top-Down Decision Tree Learning Algorithms. Proceedings of the 28th Annual ACM Symposium on the Theory of Computing, ACM Press, pages 459--468, 1996.

.... an algorithm called Adaboost was proposed that meets the formal criteria of the boosting model and is also competitive in practice [10] Second, the basic algorithms underlying the popular C4:5 and CART programs have also very recently been shown to meet the formal criteria of the boosting model [12]. Thus, it seems plausible that the weak learning framework may provide a setting for interaction between formal analysis and machine learning practice that is lacking in other theoretical models. Our aim in this paper is to push this interaction further in light of these recent developments. In ....

....recent developments. In particular, we perform experiments suggested by the formal results for Adaboost and C4:5 within the weak learning framework. We concentrate on two particularly intriguing issues. First, the theoretical boosting results for top down decision tree algorithms such as C4:5 [12] suggest that a new splitting criterion may result in trees that are smaller and more accurate than those obtained using the usual information gain. We confirm this suggestion experimentally. Second, a superficial interpretation of the theoretical results suggests that Adaboost should vastly ....

[Article contains additional citation context not shown here]

M. Kearns and Y. Mansour. On the boosting ability of topdown decision tree learning algorithms. In Proceedings of the 28th ACM Symposium on the Theory of Computing. ACM Press, New York, NY, 1996. To appear.

No context found.

Kearns, M., & Mansour, Y. (1996) On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1), 1999, pp 109128. Also in Proceedings ACM Symposium on the Theory of Computing, 1996, ACM Press, pp.459-468.

No context found.

Kearns, M. J., & Mansour, Y. (1999). On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1), 109--128.

No context found.

M. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing, pages 459-468, 1996.

No context found.

M.J. Kearns and Y. Mansour. On the Boosting Ability of Top-Down Decision Tree Learning Algorithms. Journal of Computer and Systems Sciences, 58(1):109-128, 1999.

No context found.

M.J. Kearns and Y. Mansour. On the Boosting Ability of Top-Down Decision Tree Learning Algorithms. Journal of Computer and Systems Sciences, 58(1):109-128, 1999.

No context found.

M.J. Kearns and Y. Mansour. On the Boosting Ability of Top-Down Decision Tree Learning Algorithms. Journal of Computer and Systems Sciences, 58(1):109-128, 1999.

No context found.

M.J. Kearns and Y. Mansour. On the Boosting Ability of Top-Down Decision Tree Learning Algorithms. Journal of Computer and Systems Sciences, 58(1):109-128, 1999.

No context found.

Kearns, M., Mansour, Y. On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1): 109-128, 1999.

No context found.

Keams, M. and Mansour, Y. "On the boosting ability of top-down decision tree learning algorithms" Proceedings of the Twenty-Eighth ACM Symposium on the Theory of Computing, pp. 459-468, New York, ACM Press, 1996.

No context found.

Michael Kearns and Yishay Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing, pages 459#468, Philadelphia,Pennsylvania, 1996. 40 SREERAMA K. MURTHY

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC