Abstract

Abstract. We present two new methods for obtaining generalization error bounds in a semi-supervised setting. Both methods are based on approximating the disagreement probability of pairs of classifiers using unlabeled data. The first method works in the realizable case. It suggests how the ERM principle can be refined using unlabeled data and has provable optimality guarantees when the number of unlabeled examples is large. Furthermore, the technique extends easily to cover active learning. A downside is that the method is of little use in practice due to its limitation to the realizable case. The idea in our second method is to use unlabeled data to transform bounds for randomized classifiers into bounds for simpler deterministic classifiers. As a concrete example of how the general method works in practice, we apply it to a bound based on cross-validation. The result is a semi-supervised bound for classifiers learned based on all the labeled data. The bound is easy to implement and apply and should be tight whenever cross-validation makes sense. Applying the bound to SVMs on the MNIST benchmark data set gives results that suggest that the bound may be tight enough to be useful in practice. 1

Citations

6694	Statistical Learning Theory - Vapnik - 1998
1515	A Theory of the Learnable - Valiant - 1984
1086	Making large-scale SVM learning practical - Joachims - 1999
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
185	Rademacher and gaussian complexities: Risk bounds and structural - Bartlett, Mendelson - 2002
59	On the exponential value of labeled samples - Castelli, Cover - 1995
50	PAC-Bayesian stochastic model selection - McAllester - 2003
44	A PAC-style model for learning from labeled and unlabeled data - Balcan, Blum - 2005
41	Bayesian Gaussian process models: PAC-Bayesian generalisation error bounds and sparse approximations - Seeger - 2003
34	Almost-everywhere algorithmic stability and generalization error - Kutin, Niyogi - 2002
33	Learning from a mixture of labeled and unlabeled examples with parametric side information - Ratsaby, Venkatesh - 1995
26	Beating the hold-out: Bounds for k-fold and progressive cross-validation - Blum, Kalai, et al. - 1999
17	Metric-based methods for adaptive model selection and regularization - Schuurmans, Southey
13	and C.Gentile. Improved risk tail bounds for on-line algorithms - Cesa-Bianchi - 2004
12	Learning by distances - Ben-David, Itai, et al. - 1995
11	Extensions to Metric-Based Model Selection - Bengio, Chapados - 2003
3	Co-validation: Using model disagreement to validate classification algorithms - Madani, Pennock, et al. - 2005
1	Relating the Rademacher and VC bounds - Kääriäinen - 2004
1	Practical prediction theory for classification (2003) A tutorial presented at ICML 2003. Available at http://hunch.net/ ∼ jl/projects/ prediction bounds/tutorial/tutorial.pdf - Langford