Abstract

Abstract PAC-Bayesian learning methods combine the informative priors of Bayesian methods with distribution-free PAC guarantees. Stochastic model selection predicts a class label by stochastically sampling a classifier according to a "posterior distribution " on classifiers. This paper gives a PAC-Bayesian performance guarantee for stochastic model selection that is superior to analogous guarantees for deterministic model selection. The guarantee is stated in terms of the training error of the stochastic classifier and the KL-divergence of the posterior from the prior. It is shown that the posterior optimizing the performance guarantee is a Gibbs distribution. Simpler posterior distributions are also derived that have nearly optimal performance guarantees.

Citations

556	The weighted majority algorithm - Littlestone, Warmuth - 1994
267	How to use expert advice - Cesa-Bianchi, Freund, et al. - 1997
176	Minimum complexity density estimation - Barron, Cover - 1991
112	Learning classification trees - BUNTINE - 1992
106	Adaptive game playing using multiplicative weights - Freund, Schapire - 1999
101	An Experimental and Theoretical Comparison of Model Selection Methods - Kearns, Mansour, et al. - 1999
83	Some PAC-Bayesian theorems - McAllester - 1999
76	M.K.: Using and combining predictors that specialize - Freund, Schapire, et al. - 1997
66	Complexity regularization with applications to artificial neural networks - BARRON - 1991
64	Predicting nearly as well as the best pruning of a decision tree - Helmbold, Schapire - 1995
61	PAC-Bayesian Model Averaging - McAllester - 1999
36	On pruning and averaging decision trees - Oliver, Hand - 1995
31	A fast, bottom-up decision tree pruning algorithm with near-optimal generalization - Kearns, Mansour - 1998
29	Results on learnability and the Vapnik-Chervonenkis dimension - Linial, Mansour, et al. - 1988
28	Mixing strategies for density estimation - Yang
27	Concept learning using complexity regularization - Lugosi, Zeger - 1996
23	An efficient extension to mixture techniques for prediction and decision trees - Pereira, Singer - 1999
22	A PAC analysis of a Bayesian estimator - Shawe-Taylor, Williamson - 1997
6	Adaptive estimation in pattern recognition by combining different procedures - Yang - 2000
2	Universal aggregation rules with sharp oracle inequalities - Catoni - 1999
1	Learning non-parametric densities in tyerms of finitedimensional parametric hypotheses - Yamanishi - 1992