S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilema. Neural Computation, 4(1):1--58, 1992.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....as closely as desired any function, as long as a sufficient number of parameters is included in the model. While useful because of this property, such approaches, nevertheless, also suffer from a problem known as the Bias Variance Dilemma. I will now describe the Bias Variance Dilemma following [Geman et al..1992], but in order to do so, it is first necessary to introduce a notation slightly different from the one used in Section 2.3.1. In this notation, I refer to D = x 1 ; y 1 ) x 2 ; y 2 ) x N ; y N ) as the training data set, and I refer to the induced function as f(x; D) rather ....

....too use the mean squared error function (times N) as the criterion to minimize. Unfortunately, there is a typical tradeoff between the bias and variance contributions to the estimation error: in general, a decrease in variance yields a large bias whereas a decrease in bias yields a large variance ([Geman et al..1992]) While parametric methods often suffer from a large bias term, they typically have the advantage of not having a large variance. Conversely, nonparametric approaches including feedforward neural networks do not have a large bias but suffer from large variance. Nevertheless, the bias variance ....

[Article contains additional citation context not shown here]

S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilema. Neural Computation, 4(1):1--58, 1992.

....In either case the performance on new data, that is the ability of the network to generalize, will be poor. The problem can be regarded as one of finding the optimal trade off between the high bias of a model which is too inflexible and the high variance of a model with too much freedom (Geman et al. 1992). There are two well known techniques for controlling the bias and variance of a model, known respectively as structural stabilization and regularization. The first of these involves making adjustments to the number of free parameters in the model as a way of controlling the number of degrees of ....

Geman S, Bienenstock E and Doursat R, 1992. Neural networks and the bias/variance dilema, Neural Computation 4 1--58.

....to the validation data itself, and so the final performance of the selected model should be confirmed using a third independent data set called a test set. Some theoretical insight into the problem of overfitting can be obtained by decomposing the error into the sum of bias and variance terms [Geman, et al. 1992]. A model which is too inflexible is unable to represent the true structure in the underlying density function and this gives rise to a high bias. Conversely a model which is too flexible becomes tuned to the specific details of the particular data set and gives a high variance. The best ....

Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias/variance dilema. Neural Computation, 4:1--58.

....this used to provide the greater potential for fault tolerance noted above. Optimal generalization occurs when the available complexity in the network is matched to that required to model the true function. This balancing act has been described in various ways such as the bias variance dilemma (Geman et al. 1992) and Occam s razor (Bishop 1995a) where the most common interpretation is to choose the simplest architecture that will provide a solution. One practical approach to this dilemma is to favour parsimony by restricting the complexity of the model adopted by the network. This group of techniques ....

Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias/variance dilema.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC