Citation Context

...dge often is not available or is difficult to quantify. Another approach is to use a prior to capture coarse information from the data that may be used to stabilize the estimation, as we have done in =-=[9]-=-. In practice, priors are sometimes chosen in Bayesian estimation to tame the tail of likelihood distributions so that expectations will exist when they might otherwise be infinite [22]. This mathemat...

Citation Context

...aper is demonstrating that simply averaging local probabilities/discriminants over a reasonable set of neighborhoods performs similarly to cross-validating one neighborhood size for local classifiers but without the costs or complexity of crossvalidation. This is verified for six different local classifiers on seven benchmark datasets. Further, we show how this can be interpreted as a Bayesian approach to estimating the neighborhood size, and refer to it as Bayesian neighborhoods. A secondary contribution is presenting a local version of the Bayesian quadratic discriminant analysis classifier [13], [14]. As with classical quadratic discriminant analysis (QDA) [5] the global version of this classifier has significant model bias that can be greatly reduced by applying it locally. Combined with the proposed Bayesian neighborhoods, the local Bayesian quadratic discriminant analysis (local BDA) proposed herein approximately minimizes expected misclassification error with respect to uncertainty in both the neighborhood and the locally modeled class-conditional Gaussians and is a state-of-the-art classifier that is also completely lazy. First, we review related approaches to finding neighborh...

Citation Context

...ave the useful property that the mean of a set has the minimum mean Bregman divergence to all the points in the set [4]. Recently, we generalized Bregman divergence to a functional Bregman divergence =-=[5]-=- [2] in order to show that the mean of a set of functions minimizes the mean Bregman divergence to the set of functions. The functional Bregman divergence is a straightforward analog to the vector cas...

Citation Context

...mensional points [9], [14]. Recently, we defined a functional Bregman divergence to characterize the distortion between two non-negative functions or two distributions rather than between two vectors =-=[6]-=-. The main result of this paper is showing that the mean minimizes the expected functional Bregman divergence. In addition, we state the relationship between functional Bregman divergence and the stan...

Citation Context

... the regularized linear interpolation and ridge weighted nearest-neighbor classifiers achieved zero error. (However, using a Bayesian QDA can also achieve approximately zero error for this simulation =-=[43]-=-.) 7.3 Cross-validation and Overfitting of the Regularization Parameters It is difficult to choose a set of parameters for cross-validation, with too dense a set of choices leading to overfitting, and...

Citation Context

...to select the number of optimal number of components in a mixture model is to minimise the classification error on a development set. As an alternative we investigate taking a model average (see e.g. =-=[14]-=-) over the number of components, i.e. to calculate the mean likelihood across models for a given data point. This gives uniformly better results in quiet conditions and can be interpreted as taking a ...

Citation Context

...DA (Linear Discriminant Analysis) based face recognition [3]. One assumption in LDA is that the data are linearly separable which is not always true about faces. Quadratic Discriminant Analysis (QDA) =-=[4]-=- relaxes this assumption and defines a quadratic surface to separate the classes. However, a common problem in LDA and QDA is that sufficient training samples covering all possible illuminations, expr...

Citation Context

...ditional term P (Xj+1,...,d = xnum|X1,...,j = xcat, Y = g) for the numerical features may be estimated using any known probability model estimation technique. For example, Gaussian models can be used =-=[67]-=-. The categorical class-conditional probability may also be estimated with known techniques. For example, a multinomial naive Bayes model can be easily estimated. Or, SDA could be used to model the cl...

Citation Context

...] ≡ arg min i xTi xi − 2xTi x + tr Λ + xT x ≡ arg min i ‖xi − x‖22. Thus, the k nearest neighbors to random x are the k nearest neighbors to x. If fewer than k samples are available for class g, then we use all available training samples for the class, so that number of samples used for the gth class is kg = min {∣∣{xi s.t. yi = g}∣∣, k}. Estimating the mean and covariance of a Gaussian using a small number of feature vectors can be ill-posed. Srivastava et al. addressed the problem of ill-posed Gaussian estimation by using a Bayesian estimate with a data-dependent inverted Wishart prior [9]. The Bayesian estimate is formed by marginalizing the unknown (random) Gaussian pdf over all 1The form in (5) is preferred to Ex|z[p(g|x)] since the latter reduces to maximizing p(g|z)—the very problem we are trying to avoid. Table 1. Estimation methods used to form N(x; x,Λ) and the corresponding x and Λ. method x Λ LS: rewrite p(x|x) = N (x; x,Λ) (HTH)−1HT z σ2w(HTH)−1 LMMSE : rewrite p(x|x) ≈ N (x; x,Λ) ΣHT (HΣHT + σ2wI)−1z σ2w(HTH)−1 joint Gauss: p(x|z) 4= N (x; x,Λ) ΣHT (HΣHT + σ2wI)−1z ( Σ−1 + H TH σ2w )−1 assuming x and z are jointly Gaussian Gaussians that could describe the d...