Abstract

We present an analysis of how the generalization performance (expected test set error) relates to the expected training set error for nonlinear learning systems, such as multilayer perceptrons and radial basis functions. The principal result is the following relationship (computed to second order) between the expected test set and training set errors: hE test ()i 0 hE train ()i + 2oe 2 eff p eff () n : (1) Here, n is the size of the training sample , oe 2 eff is the effective noise variance in the response variable(s), is a regularization or weight decay parameter, and p eff () is the effective number of parameters in the nonlinear model. The expectations h i of training set and test set errors are taken over possible training sets and training and test sets 0 respectively. The effective number of parameters p eff () usually differs from the true number of model parameters p for nonlinear or regularized models; this theoretical conclusion is supported by M...

Citations

1391	Robust Statistics - Huber - 1981
1131	A new look at the statistical model identification - Akaike - 1974
944	J: General Linear Models - McCullagh, Nelder - 1989
840	Information theory and an extension of the maximum likelihood principle - Akaike - 1973
190	Spline Smoothing and Nonparametric Regression - Eubank
76	Statistical predictor identification - Akaike - 1970
37	Note on generalization, regularization, and architecture selection in nonlinear learning systems - Moody - 1991
21	Predicted squared error: a criterion for automatic model selection - Barron - 1984
1	Long version of this paper - Moody - 1992