John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, (27):221--234, 1987.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....I use in this thesis creates a decision tree based on the complete training set using the formulae given above. Once the decision tree is built, C4.5 prunes the tree to avoid overfitting, again based on a user specified setting. The pruning is based on Quinlan s pessimistic error pruning (PEP) Qui87] which works as follows: For a given node, n, it considers all instances that are covered by n i.e. all the instances that have passed down from the root to n. It pessimistically estimates the error it would expect to see if n was a leaf node, based on predicting the majority ....

J. Ross Quinlan. Simplifying decision trees. International Journal of ManMachine Studies, 27:221--234, 1987.

....local information, whereas grafting uses non local information. The use of both pruning and grafting in conjunction is demonstrated to provide the best general predictive accuracy over a representative selection of learning tasks. 1 Introduction Decision tree pruning [ Breiman et al. 1984; Quinlan, 1987 ] is a widely accepted method for post processing decision trees. Pruning removes nodes from an inferred decision tree. It has been demonstrated to improve the predictive accuracy of inferred decision trees in a wide variety of domains [ Breiman et al. 1984; Quinlan, 1987 ] A classifier can be ....

....[ Breiman et al. 1984; Quinlan, 1987 ] is a widely accepted method for post processing decision trees. Pruning removes nodes from an inferred decision tree. It has been demonstrated to improve the predictive accuracy of inferred decision trees in a wide variety of domains [ Breiman et al. 1984; Quinlan, 1987 ] A classifier can be viewed as partitioning an instance space. Each partition associates a set of possible objects with a class. Pruning reduces the number of partitions imposed on an instance space by a decision tree. In contrast to pruning, a number of recent studies have suggested that ....

J. Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

.... and Kibler, 1992 ] Rule sets are also representationally more powerful than competing representations such as decision trees; this is reflected in the fact that rule induction systems significantly outperform standard decision tree induction systems on many problems [ Pagallo and Hassler, 1990; Quinlan, 1987; Weiss and Indurkhya, 1991 ] Finally, many sorts of prior knowledge about a learning problem can be communicated to a rule induction system by providing appropriate constraints on the form of induced rules [ Cohen, 1991; Cohen, 1992a ] The goal of this paper is to study the degree to which ....

....noise is added to the problem. To allow pFOIL to cope with noisy data, we added to pFOIL a variant of Brunk and Pazzani s reduced error pruning [ 1991 ] which was in turn inspired by Pagallo and Haussler s pruning algorithm for decision lists [ 1990 ] and earlier work in pruning decision trees [ Quinlan, 1987 ] The reduced error pruning algorithm consists of two phases, each of which requires an independent sample. Practically speaking, this requires dividing the training data into two sets, a primary training set and a retraining set. In all the experiments of this paper, the primary training set ....

[Article contains additional citation context not shown here]

J. Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

....the model. Such models are less efficient to store and use than their correctly sized counterparts. Using these models requires the collection of unnecessary data. Portions of overfitted models are wrong and mislead users. Finally,overfitting can reduce the accuracy of induced models on new data [8]. For induction algorithms that build decision trees [1, 7, 10] is a common approach to correct overfitting. Pruning methods take an induced tree, examine individual subtrees, and remove those subtrees deemed to be unnecessary. Pruning methods primarily differ in the criterion used to judge ....

....training set size and tree size, even after accuracy has ceased to increase. The relationship between training set size and tree size was explored with 4 pruning methods and 19 datasets taken from the UCI repository. The pruning methods are error based ( the default) 7] reduced error ( [8], minimum description length ( 9] and cost complexity with the rule ( 1] The majority of extant pruning methods take one of four general approaches: deflating accuracy estimates based on the training set (e.g. pruning based on accuracy estimates from a pruning set (e.g. managing the ....

J. Ross Quinlan. Simplifying decision trees. , 27:221--234, 1987.

....so that the final concept description does not classify all training instances correctly. Post Pruning means that first a concept description is generated that perfectly explains all training instances. This will be subsequently gener2 alized by cutting off branches of the decision tree (as in [Quinlan, 1987] or [Breiman et al. 1984] In ILP, Pre Pruning has been common in the form of stopping criteria as used in Foil [Quinlan, 1990] mFoil [Dzeroski and Bratko, 1992] and Fossil (see section 2) Post Pruning was introduced to ILP with an adaptation of Quinlan s Reduced Error Pruning [Brunk and ....

John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

....that the size of the learned concepts (and thus the amount of overfitting) may increase with training set size [Furnkranz, 1994a] 3. 2 Post Pruning Post pruning was introduced to relational learning algorithms with Reduced Error Pruning (REP) Brunk and Pazzani, 1991] based on previous work by [Quinlan, 1987] and [Pagallo and Haussler, 1990] The basic idea is that in a first pass, no attention is payed to the noise in the data and a concept description that explains all of the positive and none of the negative examples is learned. For this purpose the training set is split into two subsets: a growing ....

John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

....but also patterns arising purely by chance. Since overfitting in this sense decreases predictive accuracy, a great deal of effort has been expended in developing overfitting avoidance methods for tree induction, generally in the form of pruning strategies. These methods have been reported widely [3,5,9,11,14,15]; they have been compared empirically [10] and few researchers would now undertake decision tree induction without relying on some form of overfitting avoidance. In fact, however, if overfitting avoidance methods have improved the predictive accuracy of induced decision trees it is not because ....

....techniques. And it seems safe to predict that indiscriminate use of these techniques will sooner or later lead to performance degradation in real data problems of practical importance. 20 20 In fact, in work conducted since this article was written [17] the pruning methods of CART [3] and C4 [14] have been shown to decrease predictive accuracy in important, 9 Practical Significance This article will have served a practical purpose if, as just suggested, it causes researchers to study the conditions under which existing overfitting avoidance techniques can be expected to increase ....

[Article contains additional citation context not shown here]

J. Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

....about the relationship between complexity and predictive accuracy of classifiers, see [380] 12 Techniques that start with a sufficient partitioning and then optimize the structure (e.g. 318] can be thought of as being a converse to this approach. 21 ffl Trees to rules conversion: Quinlan [393, 398] gave efficient procedures for converting a decision tree into a set of production rules. Simple heuristics to generalize and combine the rules generated from trees can act as a substitute for pruning for Quinlan s univariate trees. ffl Other: Cockett and Herrera [90] suggested a method to reduce ....

....accuracy on a pruning set. Pruning set is a portion of the training data that is set aside exclusively for pruning alone. Use of a separate pruning set is a fairly common practice. A method other than cost complexity pruning that needs a separate pruning set is Quinlan s reduced error pruning [393]. This method, unlike cost complexity pruning, does not build a sequence of trees and hence is claimed to be faster. Chou et al. 85] extended Breiman et al. s pruning method to tree structured vector quantizers. The requirement for an independent pruning set might be problematic especially when ....

[Article contains additional citation context not shown here]

John Ross Quinlan. Simplifying decision trees. International Journal of ManMachine Studies, 27:221--234, 1987.

....so that the size of the learned concepts (and thus the amount of overfitting) may increase with training set size [Furnkranz, 1994b] 2. 2 Post Pruning Post pruning was introduced to relational learning algorithms with Reduced Error Pruning (REP) Brunk and Pazzani, 1991] based on previous work by [Quinlan, 1987] and [Pagallo and Haussler, 1990] The basic idea is that in a first pass, no attention is paid to the noise in the data and a concept description that explains all of the positive and none of the negative examples is learned. For this purpose the training set is split into two subsets: a growing ....

John Ross Quinlan. Simplifying decision trees. International Journal of ManMachine Studies, 27:221--234, 1987.

....the classification problem in detail in Section 5, and present a concrete algorithm for classification problems obtained by combining these operations. We show that the classifier so obtained is not only efficient but has a classification accuracy comparable to the well known classifier ID3 [14]. We present our conclusions and directions for future work in Section 6. 2 Database Mining Problems We present three classes of database mining problems that we have identified by examining some of the often cited applications of database mining. These classes certainly do not exhaust all ....

....paths from the root to the leaves of the classification tree. Given a string s in the seed, we Generate all extensions of this string by adding all possible (attribute, value) pairs to it. Combination is performed on new strings generated through an extension by a continuous valued attribute. ID3 [14] and CART [2] use binary splitting [13] for this purpose, whereas IC [1] partitions the domain of a continuous attribute into intervals. In the Filter operation, entropy [13] is computed for each attribute added, and only the strings containing the attribute that has the highest value of ....

[Article contains additional citation context not shown here]

J. Ross Quinlan, "Simplifying Decision Trees", Int. J. Man-Machine Studies, 27, 1987, 221--234.

....methods is to learn a concept description on one part of the training instances and to subsequently delete several parts of this theory in order to improve performance on the remaining set. The most prominent use of this method in ILP is the adaptation [3] of Reduced Error Pruning (REP) [17]. However, it has been shown in [4] that REP can be very inefficient, because most of the time is wasted for generating clauses that explain noisy examples and subsequently pruning these clauses. We attempt to solve this problem by adapting the relational learning algorithm Fossil to combine ....

....so that the final concept description does not classify all training instances correctly [16, 1] Post Pruning means that first a concept description is generated that perfectly explains all training instances. This will be subsequently generalized by cutting off branches of the decision tree [17, 2]. In ILP, Pre Pruning has been common in the form of stopping criteria based on encoding length [18] significance testing [6] or compression [14] Post Pruning was introduced to ILP with an adaptation of Quinlan s Reduced Error Pruning [3] First the training set is split into two subsets: a ....

John Ross Quinlan, `Simplifying decision trees', International Journal of Man-Machine Studies, 27, 221--234, (1987).

....systems, including C4.5 [Qui93] FOIL [Qui90] CIGOL [MB88] and GOLEM [MF90] accept as input a set of facts and output general theories. While the input generally consists of unconnected facts, the output consists of a relatively small number of highly connected generalisations. Quinlan s [Qui87] method for simplifying decision trees is another example of a policy which increases coherence. Branches in a decision tree can become highly complex and specialised, leaving the theory highly susceptible to noisy data. The process of pruning a decision tree removes such specialised branches in ....

Ross Quinlan. Simplifying decision trees. In International Journal of Man-Machine Studies 27, pages 221--234, 1987.

....or simple pruning is linear in tree height, contrasted to the exponential growth of more complex operations. The key factor that influences whether simple pruning will suffice is whether the split selection and pruning heuristics are the same and unbiased. ffl Trees to rules conversion: Quinlan [302, 306] gave efficient procedures for converting a decision tree into a set of production rules. Simple heuristics to generalize and combine the rules generated from trees can act as a substitute for pruning for Quinlan s univariate trees. ffl Tree reduction: Cockett and Herrera [61] suggested a method ....

....as the pruned tree, based on its classification accuracy on a pruning set. Pruning set is a portion of the training data that is set aside exclusively for pruning alone. Use of a separate pruning set is a fairly common practice. Another pruning method that needs a separate data set is Quinlan s [302] reduced error pruning. This method, unlike cost complexity pruning, does not build a sequence of trees and hence is claimed to be faster. The requirement for an independent pruning set might be problematic especially when small training samples are involved. Several solutions have been suggested ....

[Article contains additional citation context not shown here]

John Ross Quinlan. Simplifying decision trees. Int. J. of Man-Machine Studies, 27:221--234, 1987.

....control data set ART2 gave a smaller error as compared to Autoclass, see Table 1. 4.2 Credit Card Approval Data Base This data base contains 690 cases of credit card applications and has 15 attributes for each case. This database has been used by Ross Quinlan is his study on decision trees [Qui87] There are no missing values in the data base. 4.3 Iris Data This is the classical classification data set first used by Fisher in his analysis of linear discrimination function [Fis36] The data contains 150 cases from three species of Iris, 50 cases from each class. Each case is described by ....

Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221-- 234, 1987. 10 Afzal Upal

....that the final concept description does not classify all training instances correctly. Post Pruning means that first a concept description is generated that perfectly explains all training instances. This theory will subsequently be generalized by cutting off branches of the decision tree (as in [Quinlan, 1987] or [Breiman et al. 1984] In Inductive Logic Programming, pre pruning has been common in the form of stopping criteria as used in FOIL [Quinlan, 1990] mFOIL [D zeroski and Bratko, 1992] or FOSSIL [F urnkranz, 1994a] Post pruning was introduced to ILP with Reduced Error Pruning (REP) Brunk ....

.... Logic Programming, pre pruning has been common in the form of stopping criteria as used in FOIL [Quinlan, 1990] mFOIL [D zeroski and Bratko, 1992] or FOSSIL [F urnkranz, 1994a] Post pruning was introduced to ILP with Reduced Error Pruning (REP) Brunk and Pazzani, 1991] based on ideas by [Quinlan, 1987] and [Pagallo and Haussler, 1990] First the training set is split into two subsets: a growing set and a pruning set. A concept description explaining all of the examples in the growing set is generated with a relational learning algorithm. The resulting concept is then generalized by deleting ....

John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

....generated from a proportion of the training data. The resulting concept is then analyzed with the remaining, unseen examples and, if necessary, is generalized to improve the accuracy on these examples. In decision tree learning post pruning approaches have e.g. been used in Reduced Error Pruning [Quinlan, 1987], CART [Breiman et al. 1984] or ASSISTANT [Cestnik et al. 1987] The main concern of the research reported in this thesis will be efficient pruning methods for rule learning algorithms. In chapters 3 and 4 we will review some of the pruning methods that have been adopted for relational concept ....

....consistent concept description. The result is subsequently analyzed and (if necessary) simplified and generalized in order to increase its predictive accuracy on unseen data. Post pruning approaches have been commonly used in the decision tree learning algorithms CART [Breiman et al. 1984] ID3 [Quinlan, 1987] and ASSISTANT [Niblett and Bratko, 1986] An overview and comparison of various approaches can be found in [Mingers, 1989a] and [Esposito et al. 1993a] This chapter will first review how Reduced Error Pruning [Quinlan, 1987] can be adapted for a rule learning algorithm (section 4.1) We will ....

[Article contains additional citation context not shown here]

John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

....that the final concept description does not classify all training instances correctly. Post Pruning means that first a concept description is generated that perfectly explains all training instances. This theory will subsequently be generalized by cutting off branches of the decision tree (as in [Quinlan, 1987] or [Breiman et al. 1984] In ILP, pre pruning has been common in the form of stopping criteria as used in Foil [Quinlan, 1990] mFoil [Dzeroski and Bratko, 1992] or Fossil [Furnkranz, 1994a] Post pruning was introduced to ILP with Reduced Error Pruning (REP) Brunk and Pazzani, 1991] based ....

.... et al. 1984] In ILP, pre pruning has been common in the form of stopping criteria as used in Foil [Quinlan, 1990] mFoil [Dzeroski and Bratko, 1992] or Fossil [Furnkranz, 1994a] Post pruning was introduced to ILP with Reduced Error Pruning (REP) Brunk and Pazzani, 1991] based on ideas by [Quinlan, 1987] and [Pagallo and Haussler, 1990] First the training set is split into two subsets: a growing set and a pruning set . A concept description explaining all of the examples in the growing set is generated with a relational learning algorithm. The resulting concept is then generalized by deleting ....

John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

No context found.

John Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, (27):221--234, 1987.

No context found.

J. Ross Quinlan. Simplifying decision trees. International Journal of Man-Machine Studies, 27:221--234, 1987.

No context found.

John Ross Quinlan. Simplifying decision trees. Int. J. of Man-Machine Studies, 27:221#234, 1987.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC