Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356--372.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....creation; finding it resulted in effective ensembles when applied to either C4.5 and backpropagation algorithms. Breiman [19] showed 3 that perturbing the outputs, either by output smearing, or by output flipping, resulted in new training sets that formed effective ensembles. Raviv and Intrator [20] describe a process of noise injection, in which variable amounts of noise are added to the inputs in order to create ensemble members (although they used noise injection together with bootstrap resampling and weight regularisation) And a method termed non linear transformations [21] has been ....

Raviv, Y. and Intrator, N. (1996) Bootstrapping with noise: an effective regularization technique. Connection Science, 8, 3/4, 355-372.

....Transformations of the data A second method of creating an ensemble was also employed, this time using only the pressure data. This method, entitled Non linear transformations of the data (NLT) is particular to our laboratory, although it is related to the idea of adding noise to the inputs (Raviv and Intrator, 1996), and can be thought of as a method of preprocessing the input data. The technique involves creating new training sets from an original by means of transforming the inputs, and then constructing an ensemble which consists of a net trained on the original data, and nets trained on transformed data. ....

Raviv, Y, and Intrator, N. (1996) Bootstrapping with noise: an effective regularization technique. Connection Science, 8, 3/4, 355-372.

....nets that are trained on different training sets than through the manipulation of other neural net parameters. There are a number of alternative ways in which different training sets could be created. Techniques designed to produce different training sets include cross validation and bootstrapping (Raviv Intrator, 1996; Krogh Vedelsby, 1995) non linear transformations (Sharkey, Sharkey and Chandroth, 1996) injection of noise during training (Raviv Intrator, 1996) data from different sensors (Sharkey, Sharkey Chandroth, 1996) the boosting algorithm (Drucker et al., 1994) and the use of different ....

....which different training sets could be created. Techniques designed to produce different training sets include cross validation and bootstrapping (Raviv Intrator, 1996; Krogh Vedelsby, 1995) non linear transformations (Sharkey, Sharkey and Chandroth, 1996) injection of noise during training (Raviv Intrator, 1996); data from different sensors (Sharkey, Sharkey Chandroth, 1996) the boosting algorithm (Drucker et al., 1994) and the use of different methods of preprocessing. Cross validation and bootstrapping both involve taking overlapping subsamples of a data set. Nonlinear transformations, and injection ....

[Article contains additional citation context not shown here]

Raviv, Y, and Intrator, N. (1996) Bootstrapping with noise: an effective regularization technique. Connection Science, 8, 3/4, 355-372.

....(1) good (not necessarily excellent) individual performance and (2) small residual error correlations with other members. Many techniques have been proposed to reduce residual correlations between members. These include resampling the training and validation data [3] adding randomness to data [7], and decorrelation training [8] These approaches are only effective for certain models and problems. Genetic algorithms have also been used to generate good and diverse members [6] Input feature selection is one of the most important stages of the model learning process. It has a crucial ....

Y. Raviv and N. Intrator. Bootstrapping with noise: An effective regularization technique. Connection Science, 8(3-4):355--72, 1996.

....They compared ensembles of 11 classifiers to a single run of FOIL and found statistically significant improvements in 15 out of 29 tasks and statistically significant loss of performance in only one task. They obtained similar results using 11 fold cross validation to construct the training sets. Raviv and Intrator (1996) combine bootstrap sampling of the training data with injecting noise into the input features for the learning algorithm. To train each member of an ensemble of neural networks, they draw training examples with replacement from the original training data. The x values of each training example are ....

Raviv, Y., & Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, 8 (3--4), 355--372.

....They compared ensembles of 11 classifiers to a single run of FOIL and found statistically significant improvements in 15 out of 29 tasks and statistically significant loss of performance in only one task. They obtained similar results using 11 fold cross validation to construct the training sets. Raviv and Intrator (1996) combine bootstrap sampling of the training data with injecting noise into the input features for the learning algorithm. To train each member of an ensemble of neural networks, they draw training examples with replacement from the original training data. The x values of each training example are ....

Raviv, Y., & Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, 8 (3--4), 355--372.

.... of choosing a diverse set of agents (i.e. uncorrelated agents) as opposed to a set of identical or highly similar agents in the averaging or weighted averaging schemes has been justified theoretically on the basis of bias variance decomposition (see e.g. Breiman 1996c, Ueda and Nakano 1996, Raviv and Intrator 1996, and so on) The heuristics of creating independent agents has been embedded in a number of well known approaches, such as bagging , in which diversity is achieved through repeated random re sampling of the training data set and the use of unstable (easily varied) agents (Breiman 1996a) and ....

....probabilities instead of as soft boundaries) Due to this difficulty, on line optimization of hard partitioning was not adopted in this work. 8 Comparisons With regard to averaging or weighted averaging, in addition to various theoretical analyses mentioned earlier (such as Breiman 1996 a, b, Raviv and Intrator 1996, Uedo and Nakano 1996) empirically, there have been demonstrations of performance advantages resulting from combining a set of (diversified) learners, for example, Hashem (1993) Perrone (1993) Parmanto, Munro and Doyle (1996) Rosen (1996) Tumer and Ghosh (1996) and Taniguchi and Tresp ....

[Article contains additional citation context not shown here]

Y. Raviv and N. Intrator, (1996). Bootstrapping with noise: an effective regularization technique.

.... classifiers could be done by either using different training heuristics (e.g. initializing with different connection weights (Battiti Colla 1994; Hansen Salamon 1990) different network architectures, different stopping and pruning criteria) and or bootstrapping the training data with noise (Raviv Intrator 1995) or without noise (Breiman 1994) It is believed that many classification models from the neural and statistics communities will potentially be a rich source to generate multiple heterogeneous classifiers. They could be easily combined via a special perceptron as described in the next section. ....

Raviv, Y. and Intrator, N. 1995. "Bootstrapping with Noise: An Effective Regularization Technique," Technical Report, Tel-Aviv University.

....obtaining confidence intervals (Baxt and White, 1995) and improved performance (Breiman, 1992; Breiman, 1994; LeBlanc and Tibshirani, 1994) of learning networks. Smooth bootstrap (Efron and Tibshirani, 1993) can also increase the independence among predictors for the purpose of ensemble averaging (Raviv and Intrator, 1996). Such methods lead to a reduction in the variance portion of the error, with little or no effect on the bias of the predictor. One can control the variance portion of the error also by imposing global assumptions about the nature of the predictor that is to be learned. These include smoothness ....

Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356--372.

....by the experts. In artificial neural networks, the independence is enhanced due to non identifiability of their models, i.e. the existence of many (local) minima of the error surface. This independence reduces the contribution of the variance portion of the error when ensemble average is used (Raviv and Intrator, 1996). An ensemble classification rule is based on averaging the real values of the outputs of all the ensemble members and then producing a decision by Bayesian classification rule. The reduction in the variance portion of the error is greater when the predictors are independent. One conventional way ....

....from the original data, however, the smooth bootstrap which adds noise to the original data leads to further independent networks and a better ensemble behavior. For example, the two spiral problem can be solved using conventional neural network and strong injection of noise into the inputs (Raviv and Intrator, 1996). 2 We take this approach a step further and add blurred images to the training set. It appears that when a single form of blur (Gaussian blur) is added to the training data, it is sufficient to improve the (test) performance on other blur operations as well. However, when training with blurred ....

Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356--372.

....Regularized Neural Network Ensembles Another way to assess the effect of regularization constraints is by combining regularized networks into ensembles. It is well known that an ensemble of experts is capable of improving the performance of single experts (Wolpert, 1992; Krogh and Vedelsby, 1995; Raviv and Intrator, 1996). There are two main questions to be addressed when constructing ensembles: i) how to evaluate an ensemble classification prediction from predictions of its members and (ii) which networks to combine. There are different ways to evaluate an ensemble classification prediction. The first, is using ....

....of experts. Thus, we present results with a simple ensemble averaging. The improvement in regression ensembles depends on the level of independence of the errors made by the experts. This independence reduces the contribution of the variance portion of the error when ensemble average is used (Raviv and Intrator, 1996). This also gives some hints which networks to combine. We consider three types of simple regression ensembles Ens. A,B and C, with network outputs averaged over: A) i values close to the optimal i , all networks are trained July 31, 2000 Stainvas et al. 12 from the same initial weights ....

[Article contains additional citation context not shown here]

Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356--372.

....amounts to generating larger datasets by simulating the true noise in the data. It was recently shown that noise added to the input during training can be viewed as a regularizing parameter that controls, in conjunction with ensemble averaging, the capacity and the smoothness of the estimator (Raviv and Intrator, 1996). The major role of this noise is to push different estimators to different local minima, and by that, produce a more independent set of estimators. Best performance is then achieved by averaging over the estimators. For this regularization, the level of the noise may be larger than the true ....

Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. To Appear: Connection Science, Special issue on Combining Estimators.

No context found.

Chapman-Hall. Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique.

....Hornik et al. 1993) These two properties of ANN make them natural candidates for modeling data. The large flexibility provided by neural network models results in prediction with a relatively small bias, but a large variance. Careful methods for variance control (Barron, 1991; Breiman, 1996; Raviv and Intrator, 1996) can lead to a smaller prediction error and are required to robustify the prediction. While artificial neural networks have been extensively studied and used in classification and regression problems, their interpretability still remains vague. The aim of this paper is to present a method for ....

....set, different local minima are found when starting from different random initial conditions. These different local minima lead to somewhat independent predictors, and thus, the averaging can reduce the variance. Hansen and Salamon, 1990; Wolpert, 1992; Perrone and Cooper, 1993; Breiman, 1996; Raviv and Intrator, 1996) When a larger set of independent networks is needed, but only little data is available, data reuse methods can be helpful for robustiying the results. Resampling (with return) from the training data improves the independence of the training sets, and hence, the independence of the estimators, ....

[Article contains additional citation context not shown here]

Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356--372.

....use of training data becomes essential. Methods for data reuse such as cross validation (Stone, 1974) and bootstrap (Efron and Tibshirani, 1993) can help in obtaining confidence intervals (Baxt and White, 1995) and improved performance (Breiman, 1992; Breiman, 1994; LeBlanc and Tibshirani, 1994; Raviv and Intrator, 1995). Unlike data, class labels are not often reused (see however, Grossman and Lapedes, 1993) in particular, multiple class labels. Humans make natural and extensive use of the fact that objects may have several class associations (say, at different category levels) In contrast, in machine ....

Raviv, Y. and Intrator, N. (1995). Bootstrapping with noise: An effective regularization technique. Preprint.

....of each predictor too much. Another approach is to bootstrap several training sets with a small percentage of non overlapping patterns (Efron and Tibshirani, 1993) A recently proposed method increases independence between the predictors by adding large amounts of noise to the training patterns (Raviv and Intrator, 1996). A different approach to ensemble averaging is the adaptive mixture of experts (Jacobs et al. 1991) This method is a divide and conquer algorithm which co trains a gating network for (soft) partitioning the input space and expert networks modeling the underlying function in each of these ....

Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique.

No context found.

Press. Raviv, Y. and Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356--372.

No context found.

Y. Raviv and N. Intrator, "Bootstrapping with noise: An effective regularization technique," Connection Science, Special issue on Combining Estimators, vol. 8, pp. 356--372, 1996.

....in the constructed classifier. Notice that this condition corresponds to the requirement mentioned earlier of maximum independence among the experts. Several ways were suggested for making the experts less dependent, one example is to inject noise during the training, as in Smooth Bootstrap [24]. Since the search for an optimal classifier is tied with the search for an optimal data representation (i.e. an optimal transformation of the input signals w.r.t the classification task at hand) it is advisable to examine and possibly use more than one signal representation. In order to ....

Y. Raviv and N. Intrator, "Bootstrapping with noise: An effective regularization technique," Connection Science, vol. 8, pp. 355--372, 1996.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC