Breiman L (1999) Combining predictors. In: Sharkey AJC (ed) Combining artificial neural nets: ensemble and modular multinet systems. Springer, Berlin Heidelberg New York, pp 31-50 121  Home/Search   Document Not in Database   Summary   Related Articles   Check

....and Applications special issue on Fusion of Multiple Classifiers. classifier outputs have been proposed [15, 17, 19, 24, 29] Furthermore, promoting diversity among classifiers prior to combining forms the basis of many strategies, including bagging, arcing, boosting and correlation control [6, 31]. Approaches to pooling classifiers can be separated into two main categories: i) simple combiners, e.g. voting [3] Bayesian based weighted product rule [22] or averaging [24, 30] and, ii) meta learners, such as arbitration [7] or stacking [34] The simple combining methods are best suited ....

L. Breiman. Combining predictors. In A. J. C. Sharkey, editor, Combining Artificial Neural Nets, pages 31--50. Springer-Verlag, 1999.

....h. Decision trees were implemented following the classic reference [6] Using unpruned trees avoids the need of introducing the regularization metaparameter in the system; moreover, maximal trees give best results with boosting when there is enough inter action between variables, as discussed in [8, 20, 23]. The model error e defined in AdaBoost (Box 1) does not differentiate the costs of misclassification for different output classes. In Box 2 we introduce a variant of AdaBoost (Sensitivity specificity Tuning Boosting: SSTBoost) which takes into account costs at two different levels. 6 Merler et ....

L. Breiman, "Combining predictors," in Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems (A. Shaxkey, ed.), (London), Springer-Verlag, 1999. pages 31-50.

....of a similar value. These two findings can be explained by the following 1. The Bootstrap method creates data subsets by small variations of the original data set. Consequently, the variations in complexity among such data subsets can be expected to be small. In fact, as pointed out by Breiman [1], unstable classifiers are necessary to exploit effectively the low diversity of the data subsets generated by the Bootstrap method. Neural networks are examples of such unstable classifiers, and, curiously, we rely on their ability to overtrain. 2. Differently, the Random Subspace method usually ....

L. Breiman. Combining predictors. In A.J.C. Sharkey, editor, Combining Artificial Neural Nets, pages 31-50. Springer-Verlag, London, 1999.

.... Decision trees were implemented following the classic reference [6] Using unpruned trees avoids the need of introducing the regularization metaparameter in the system; more over, maximal trees give best results with boosting when there is enough inter action between variables, as discussed in [8, 20, 23]. The model error e defined in AdaBoost (Box 1) does not differentiate the costs of misclassification for different output classes. In Box 2 we introduce a variant of AdaBoost (Sensitivity specificity Tuning Boosting: SSTBoost) which takes into account costs at two different levels. Firstly, ....

L. Breiman, "Combining predictors," in Combining Artificial Neural Nets: Ensem- ble and Modular Multi-Net Systems (A. Sharkey, ed.), (London), Springer-Verlag, 1999. pages 31-50.

....aggregating . From Adaptive reweighting and combining . 2 Bagging and Boosting 2. 1 Bagging Bagging and Boosting are strategies for creating classi er ensembles, similar by the concept, yet with fundamental di erences [9] Bagging was proposed by Breiman [2] and extended further to Arcing [3,4] to accommodate the adaptive incremental construction of the ensemble which underlies the Boosting method (explained later) Bagging creates the classi ers in the ensemble by taking random samples with replacement (bootstrap sampling [7] from the data set and building one classi er on each ....

L. Breiman. Combining predictors. In A.J.C. Sharkey, editor, Combining Arti cial Neural Nets, pages 31-50. Springer-Verlag, London, 1999.

....of diversity being used. These measures aim to quantify the dependence between classi ers. Several techniques exist which aim to improve the performance of classi er ensembles by manipulating the data set which classi ers are trained on. These include Bagging, Boosting, and Arcing [2, 3]. We are particularly interested in the AdaBoost 1 algorithm which has had considerable success with arti cial and real world data problems [1] AdaBoost builds a classi er ensemble by starting with one classi er and adding new classi ers, one at a time. The new classi er is constructed ....

....success with arti cial and real world data problems [1] AdaBoost builds a classi er ensemble by starting with one classi er and adding new classi ers, one at a time. The new classi er is constructed by modifying the training set according to the previous classi er s performance [2,4,18]. It has been found that Boosting can be paralysed [21] i.e. no further improvement is achieved when adding new classi ers to the team. Here we study how diversity changes in the ensemble as the number of classi ers produced by AdaBoost increases and try to establish whether or not ....

[Article contains additional citation context not shown here]

L. Breiman. Combining predictors. In A. J. C. Sharkey, editor, Combining Arti- cial Neural Nets, chapter 2, pages 31-50. Springer-Verlag, 1999.

....the previously described bias variance decomposition cannot be applied. A number of alternative bias variance decompositions for this case have been described in the literature. In particular, Breiman describes a decomposition in which the role of the variance is played by a term named spread [7]. In the same paper the author showed that the spread can dramatically be reduced by bagging, a committee approach introduced in Section 2.3. 4 2.2 Simple Averaging and Simple Voting In this approach, committee members are typically neural networks. The neural networks are all trained on the ....

....procedure is disturbed such that the correlation between the estimators is reduced, the generalization performance of the combined systems can further be improved. The most important representative of this approach was introduced by Breiman under the name of bagging (bootstrap aggregation) [7]. The idea behind bagging can be understood in the following way. Let s assume that each committee member is trained on a di erent data set. Then, surely, the covariance between the predictions of the individual members is zero. Unfortunately, we typically have to work with a xed training data ....

[Article contains additional citation context not shown here]

Breiman, L., Combining predictors, in Combining articial neural nets, Sharkey, A. J. C., Ed., Springer, 31, 1999.

....the predictor developed in the first round have a greater probability of being included in the next training set. The process continues for the specified number of rounds. Ensembles created through Adaboost have been shown to produce good results when compared to Bagging on a number of data sets ([23]) Breiman suggests that bagging, and some of the methods described above as distortion (namely randomising the construction of predictors, and randomising the outputs) are essentially cut from the same cloth , and that there is something fundamentally different about the adaptive resampling ....

Breiman, L. (1999) Combining predictors. In A.J.C. Sharkey (Ed) Combining Artificial Neural Nets: Ensemble and Modular Multi-net Systems. London, Springer-Verlag, pp 31-50.

....difficult for prior experts to learn. In classification, the ensemble output is usually a weighted majority of expert outputs. 1 The terms boosting and arcing are often used synonymously to describe algorithms which build ensembles by training experts on adaptively reweighted training data [5]. It has been suggested [10] that the term boosting be reserved for algorithms which can provably be shown to produce arbitrarily accurate learners; since no such proof exists for POCA, we use the term arcing. 2 In this paper, we use the term online to refer to the situation where an ....

....properties [2; 3] Recently, Breiman introduced Arc x4, a similar but simpler algorithm [4] Unlike Adaboost, Arc x4 assigns fixed, equal voting weights for each expert, and reweights exemplars using a much simpler function of the number of mistakes made by prior experts. Importantly, Breiman [5] has demonstrated that Arc x4 usually performs as well as Adaboost, and has argued that the success of arcing algorithms is due to the general process of increasing weights on difficult exemplars, rather than to any specific reweighting or recombination scheme. Pseudo code for Arc x4 is shown in ....

[Article contains additional citation context not shown here]

Breiman, L. 1999. Combining Predictors. In Combining Artificial Neural Nets. ed. A. J. C. Sharkey, 31-50. London: Springer-Verlag.

....data analysis from homogeneous data sites involves combining di erent models of data from di erent sites. There exist several techniques to combine multiple data models. A class of statistical techniques for aggregating multiple models generated from homogeneous data sets is proposed in [2, 3]. The Bagging approach [2] increases the accuracy of the model by generating multiple models from di erent data sets chosen uniformly with replacement and then averaging the outputs of the models. Although the Bagging technique was initially developed for increasing the accuracy of unstable data ....

....and then averaging the outputs of the models. Although the Bagging technique was initially developed for increasing the accuracy of unstable data analysis algorithms, this can be extended to multiple model aggregation in DDM from homogeneous data sets. This perspective is presented in [3]. Like Bagging, Stacking o ers an alternate technique to increase the accuracy of the data model by aggregating multiple data models. In Stacking rst multiple models are learned on di erent homogeneous data sets and their joint generalization behavior is observed on a di erent testing data set. ....

L. Breiman. Combining predictors. In Combining Articial Neural Nets, pages 31-50. Springer-Verlag, 1999.

....note with these results is that the actual accuracy figures are very poor. Ironically, the disagreement of individual networks is a key requirement for improving performance by building an ensemble of networks and aggregating the results of these nets to produce improved results. Breiman has shown [6] that for unstable predictors, aggregating the output of several models will reduce variance and give more accurate predictions. Evidently there is no advantage in aggregating the results of a committee of experts if they all agree. In the final section of this paper an ensemble solution to the ....

Breiman L., Combining Predictors, in Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems, A.J.C. Sharkey (ed.) pp 31-50, Springer, London, 1999.

....have explored the use of a tree based classifier for f m (x) and have demonstrated that it consistently produces significantly lower error rates than a single decision tree. In fact, Breiman (NIPS workshop, 1996) called AdaBoost with trees the best off the shelf classifier in the world (see also Breiman (1998)) Interestingly, the test error seems to consistently decrease and then level off as more classifiers are added, rather than ultimately increase. For some reason, it seems that AdaBoost is immune to overfitting. Figure 1 shows the performance of Discrete AdaBoost on a synthetic classification ....

Breiman, L. (1998), Combining predictors, Technical report, Statistics Department, University of California, Berkeley.

....have explored the use of a tree based classifier for f m (x) and have demonstrated that it consistently produces significantly lower error rates than a single decision tree. In fact, Breiman (NIPS workshop, 1996) called AdaBoost with trees the best off the shelf classifier in the world (see also Breiman (1998)) Interestingly, the test error seems to consistently decrease and then level off as more classifiers are added, rather than ultimately increase. For some reason, it seems that AdaBoost is immune to overfitting. Figure 1 shows the performance of Discrete AdaBoost on a synthetic classification ....

Breiman, L. (1998), Combining predictors, Technical report, Statistics Department, University of California, Berkeley.

No context found.

Breiman L (1999) Combining predictors. In: Sharkey AJC (ed) Combining artificial neural nets: ensemble and modular multinet systems. Springer, Berlin Heidelberg New York, pp 31-50 121

No context found.

L. Breiman, "Combining predictors", Combining artificial neural nets, A. Sharkey (Ed.), Springer-Verlag, London, 1999.

No context found.

Breiman L. Combining predictors. In: Sharkey AJC (ed), Combining Artificial Neural Nets. Springer-Verlag, 1999; 31--50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC