Abstract

In this paper, on the one hand, we aim to give a review on literature dealing with the problem of supervised learning aided by additional unlabeled data. On the other hand, being a part of the author's first year PhD report, the paper serves as a frame to bundle related work by the author as well as numerous suggestions for potential future work. Therefore, this work contains more speculative and partly subjective material than the reader might expect from a literature review. We give a rigorous definition of the problem and relate it to supervised and unsupervised learning. The crucial role of prior knowledge is put forward, and we discuss the important notion of input-dependent regularization. We postulate a number of baseline methods, being algorithms or algorithmic schemes which can more or less straightforwardly be applied to the problem, without the need for genuinely new concepts. However, some of them might serve as basis for a genuine method. In the literature revi...

Citations

6694	Statistical Learning Theory - Vapnik - 1998
6231	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
3909	Neural Networks for Pattern Recognition - Bishop - 1995
2801	Optimization by Simulated Annealing - Kirkpatrick, Gelatt, et al. - 1983
1655	A tutorial on support vector machines for pattern recognition - Burges - 1998
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
1086	Making large-scale SVM learning practical - Joachims - 1999
989	Spline models for observational data - Wahba - 1990
946	Combining labeled and unlabeled data with co-training - Blum, Mitchell - 1998
917	Statistical decision theory and Bayesian analysis - Berger - 1985
896	BD: Pattern Recognition and Neural Networks - Ripley - 1996
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
687	Estimation of dependences based on empirical data - Vapnik - 1982
634	Hierarchical mixtures of experts and the EM algorithm - Jordan, Jacobs - 1994
632	Text classification from labeled and unlabeled documents using - Nigram, McCallum, et al.
612	A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998
595	Finite Mixture Models - McLachlan, Peel - 1997
561	Y: Improved boosting algorithms using confidence-rated predictions - RE, Singer - 1999
511	DM: Statistical Analysis of Finite Mixture Distributions Chichester - Titterington - 1985
507	An Introduction to Computational Learning Theory - Kearns, Vazirani - 1994
469	editor. Learning in Graphical Models - Jordan - 1998
425	Mixture densities, maximum likelihood, and the EM algorithm - Redner, Walker - 1984
416	Bayesian classification (AutoClass): Theory and results - Cheeseman, Stutz - 1996
402	Active learning with statistical models - Cohn, Ghahramani, et al. - 1996
359	Unsupervised models for named entity classification - Collins, Singer - 1999
347	B: Nonparametric Regression and Generalized Linear Modes: A Roughness Penalty Approach Chapman and Hall - Green, Silverman - 1994
334	Mixtures of probabilistic principal component analyzers - Tipping, Bishop - 1999
328	The information bottleneck method - Tishby, Pereira, et al. - 1999
315	Exploiting generative models in discriminative classifiers - Jaakkola, Haussler - 1998
281	Convolutional kernels on discrete structures - Haussler - 2007
256	Selective sampling using the query by committee algorithm, Machine Learning 28 - Freund, Seung, et al. - 1997
247	Principal curves - Hastie, Stuetzle - 1989
243	Smopolinsky H: Query by committee - Seung, Opper - 1992
234	GTM: The Generative Topographic Mapping - Bishop, Svensen, et al. - 1998
198	Employing EM and pool-based active learning for text classification - McCallum, Nigam - 1998
197	G: The EM algorithm for mixtures of factor analyzers - Ghahramani, Hinton - 1997
191	Elements of Information Theory. Wiley series in telecommunications. Wiley-Interscience - Cover, Thomas - 1991
181	Gaussian processes for regression - Williams, Rasmussen - 1996
162	Differential-Geometrical Methods in Statistics - Amari - 1985
160	Prediction with Gaussian processes: From linear regression to linear prediction and beyond - Williams - 1998
157	Supervised learning from incomplete data via an EM approach - Ghahramani, Jordan - 1994
146	Semi-supervised support vector machines - Bennet, Demiriz - 1998
133	Bayesian Methods for Adaptive Models - MacKay - 1992
125	Using the Fisher kernel method to detect remote protein homologies - Jaakkola, Diekhans, et al. - 1999
120	M.: Variational inference for Bayesian mixtures of factor analysers - Ghahramani, Beal - 2000
117	Agglomerative information bottleneck - Slonim, Tishby - 1999
116	Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature 355 - Becker, GE - 1992
116	Information geometry and alternating minimization procedures - Csiszár, Tusnady - 1984
113	An Introduction to Latent Variable Models - Everitt - 1984
109	Is learning the n-th thing any easier than learning the first - Thrun - 1996