Abstract Three estimates of the leave-one-out error for *-support vector (SV) machine binary classifiers are presented. Two of the estimates are based on the geometrical concept of the span, which was introduced in the context of bounding the leave-one-out error for C-SV machine binary classifiers, while the third is based on optimisation over the criterion used to train the *-support vector classifier. It is shown that the estimates presented herein provide informative and efficient approximations of the generalisation behaviour, in both a toy example and benchmark data sets. The proof strategies in the *-SV context are also compared with those used to derive leave-one-out error estimates in the C-SV case 1 1 Introduction The estimation of the generalisation performance of support vector machine classifiers is an important and ongoing area of research. In the absence of a large body of data to be used for validation, it becomes necessary to estimate generalisation error using approximations that depend in some way on the training data. One such estimate is the leave-one-out error. In this case, a single point is excluded from the training set, and the classifier is trained using the remaining points. It is then determined whether this new classifier correctly labels the point that was excluded. The process is repeated over the entire training set, and the leave-one-out error is computed by taking the average over these trials; this provides an almost unbiased estimate of the generalisation error. One shortcoming of the leave-one-out method is that it is highly inefficient. To estimate the error, it is necessary to re-train the classifier over every support vector in the training set (non-support vectors do not contribute to the form of the final classifier, and can be excluded with impunity). Even assuming that the classifier is sparse in the training data, the computational overhead remains substantial. Thus methods are sought to speed calculation of the leave-one-out error, or bound it with an easily computed quantity.

We propose an algorithm to predict the leave-one-out (LOO) error for kernel based classifiers. To achieve this goal with computational efficiency, we cast the LOO error approximation task into a classification problem. This means that we need to learn a classification of whether or not a given training sample - if left out of the data set - would be misclassified. For this learning task, simple data dependent features are proposed, inspired by geometrical intuition. Our approach allows to reliably select a good model as demonstrated in simulations on Support Vector and Linear Programming Machines. Comparisons to existing learning theoretical bounds, e.g. the span bound, are given for various model selection scenarios.

One of the major obstacles to using Bayesian methods for pattern recognition has been its computational expense. This thesis presents an approximation technique that can perform Bayesian inference faster and more accurately than previously possible. This method, "Expectation Propagation," unifies and generalizes two previous techniques: assumeddensity filtering, an extension of the Kalman filter, and loopy belief propagation, an extension of belief propagation in Bayesian networks. The unification shows how both of these algorithms can be viewed as approximating the true posterior distribution with a simpler distribution, which is close in the sense of KL-divergence. Expectation Propagation exploits the best of both algorithms: the generality of assumed-density filtering and the accuracy of loopy belief propagation. Loopy belief propagation, because it propagates exact belief states, is useful for limited types of belief networks, such as purely discrete networks. Expectation Propagati...

We propose a meta learning framework, casting leave-one-out (LOO) error approximation into a classification problem. For Support Vector Machines this means that we need to learn a classification of whether or not a given Support Vector -- if left out of the data set -- would be misclassified. For this learning task, simple data set dependent features are proposed, inspired by bounds from learning theory and geometrical intuition. Our approach allows to predict the LOO error on unseen data with an astonishing degree of accuracy -- as demonstrated in simulations.

We present a new local strategy to solve incremental learning tasks. Applied to Support Vector Machines based on local kernel, it allows to avoid re-learning of all the parameters by selecting a working subset where the incremental learning is performed. Automatic selection procedure is based on the estimation of generalization error by using theoretical bounds that involve the margin notion. Experimental simulation on three typical datasets of machine learning give promising results.

We develop a fully-automatic pattern search methodology for model selection of support vector machines (SVMs) for regression and classification.

Abstract—Error-correcting output coding (ECOC) is a widely used multicategory classification algorithm that decomposes multiclass problems into a set of binary classification problems. In this paper, we propose a new method based on a bi-classification strategy, consisting of one-vs-one and ECOC classification. Also we introduce methods to improve a standard ECOC. The proposed method is compared to other algorithms by performing experiments with gene expression datasets.