• Summary
• Related Documents
  • Active Bibliography
  • Co-citation
• Version History

Abstract:

Abstract--- This article exposes problems of the commonly used technique of splitting the available data into training, validation, and test sets that are held fixed, warns about drawing too strong conclusions from such static splits, and shows potential pitfalls of ignoring variability across splits. Using a bootstrap or resampling method, we compare the uncertainty in the solution stemming from the data splitting with neural network specific uncertainties (parameter initialization, choice of number of hidden units, etc.). We present two results on data from the New York Stock Exchange. First, the variation due to different resamplings is significantly larger than the variation due to different network conditions. This result implies that it is important to not over-interpret a model, or an ensemble of models, estimated on one specific split of the data. Second, on each split, the neural network solution with early stopping is very close to a linear model; no significant nonlinearities are extracted. The data set used in this article is available from the web sites of the authors. Keywords---Model evaluation. Bootstrap. Resampling. Financial forecasting. Time series prediction. Linear bias of early stopping. Superposition of forecasts. Model merging. I.

Citations