Abstract

In this paper, on the one hand, we aim to give a review on literature dealing with the problem of supervised learning aided by additional unlabeled data. On the other hand, being a part of the author's first year PhD report, the paper serves as a frame to bundle related work by the author as well as numerous suggestions for potential future work. Therefore, this work contains more speculative and partly subjective material than the reader might expect from a literature review. We give a rigorous definition of the problem and relate it to supervised and unsupervised learning. The crucial role of prior knowledge is put forward, and we discuss the important notion of input-dependent regularization. We postulate a number of baseline methods, being algorithms or algorithmic schemes which can more or less straightforwardly be applied to the problem, without the need for genuinely new concepts. However, some of them might serve as basis for a genuine method. In the literature revi...

Citations

6696	Statistical Learning Theory - Vapnik - 1998
6236	Maximum likelihood from incomplete data via EM algorithm - Dempster, Laird, et al. - 1977
3913	CM: Neural Networks for Pattern Recognition - Bishop - 1995
2802	Optimization by Simulated Annealing - Kirkpatrick, Gelati, et al. - 1983
1656	CJC: A Tutorial on Support Vector Machines for Pattern Recognition - Burges - 1998
1326	Schapire R: Experiments with a new boosting algorithm - Freund
1085	Making large-Scale SVM learning practical - Joachims - 1999
989	Spline models for observational data - Wahba - 1990
947	Combining Labeled and Unlabeled Data with Co-training - Blum, Mitchell - 1998
917	Statistical decision theory and Bayesian analysis - Berger - 1985
896	Pattern Recognition and Neural Networks - Ripley - 1996
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
687	Estimation of dependences based on empirical data - Vapnik - 1982
634	Hierarchical mixtures of experts and the EM algorithm - Jordan, Jacobs - 1994
633	T (2000) Text classification from labeled and unlabeled documents using EM - Nigam, McCallum, et al.
612	A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998
595	Finite Mixture Models - McLachlan, Peel - 2000
561	Improved boosting algorithms using confidence-rated predictions - Schapire, Singer - 1999
510	Statistical Analysis of Finite Mixture Distributions - Titterington, Smith, et al. - 1985
507	An Introduction to Computational Learning Theory - Kearns, Vazirani - 1994
469	Learning in graphical models - Jordan - 1998
425	Mixture densities, maximum likelihood, and the EM algorithm - Redner, Walker - 1984
416	Bayesian classification (AutoClass): Theory and results - Cheeseman, Stutz - 1996
402	Active learning with statistical models - Cohn, Ghahramani, et al. - 1996
359	Unsupervised Models for Named Entity Classification - Collins, Singer - 1999
347	B: Nonparametric Regression and Generalized Linear Modes: A Roughness Penalty Approach Chapman and Hall - Green, Silverman - 1994
334	Mixtures of probabilistic principal component analyzers - Tipping, Bishop - 1999
328	The information bottleneck method - Tishby, Pereira, et al. - 1999
315	Exploiting generative models in discriminative classifiers - Jaakkola, Haussler - 1998
281	Convolutional kernels on discrete structures - Haussler - 2007
256	Selective sampling using the query by committee algorithm, Machine Learning 28 - Freund, Seung, et al. - 1997
247	Principal curves - Hastie, Stuetzle - 1989
243	Smopolinsky H: Query by committee - Seung, Opper - 1992
235	GTM: the generative topographic mapping - Bishop, Svensen, et al. - 1998
198	Employing EM and pool-based active learning for text classification - McCallum, Nigam - 1998
197	G: The EM algorithm for mixtures of factor analyzers - Ghahramani, Hinton - 1997
191	Elements of Information Theory. Wiley series in telecommunications. Wiley-Interscience - Cover, Thomas - 1991
181	Gaussian processes for regression - Williams, Rasmussen - 1996
163	Differential-Geometrical methods in statistics - Amari - 1985
160	Prediction with Gaussian processes: From linear regression to linear prediction and beyond - Williams - 1998
157	Supervised learning from incomplete data via an EM approach - Ghahramani, Jordan - 1994
147	Semi-supervised support vector machines - Bennett, Demiriz - 1998
133	Bayesian Methods for Adaptive Models - MacKay - 1992
126	Using the Fisher Kernel method to detect remote protein homologies - Jaakkola, Diekhans, et al. - 1999
120	M.: Variational inference for Bayesian mixtures of factor analysers - Ghahramani, Beal - 2000
117	Agglomerative information bottleneck - Slonim, Tishby - 1999
116	Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature 355 - Becker, GE - 1992
116	Information geometry and alternating minimization procedures - Csiszár, Tusnady - 1984
113	An Introduction to Latent Variable Models - Everitt - 1984
109	Is learning the n-th thing any easier than learning the first - Thrun - 1996