Y. Lee and O. L. Mangasarian. RSVM: reduced support vector machines. In CD Proceedings of the First SIAM International Conference on Data Mining, Chicago, 2001.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....the idea of solving approximations to RLSC. Hopefully, these approximations will yield a vast increase in speed with very little loss in accuracy. Several authors have suggested the use of low rank approximations to the kernel matrix in order to avoid explicit storage of the entire kernel matrix [19, 26, 20, 27, 22]. These techniques can be used in a variety of methods, including Regularized Least Squares Classification, Gaussian Process regression and classification, and interior point approaches to Support Vector Machines. The approaches all rely on choosing a subset of m of the training points (or a ....

Y.-J. Lee and O. Mangasarian, Rsvm: Reduced support vector machines, In SIAM International Conference on Data Mining (2001).

....hand, if the linear kernel (i.e. K(x i , x j ) x i x j ) is used, Q is directly the product of a rectangular matrix and its transpose so we can exploit more possibilities to design efficient algorithms. Thus, here we mainly consider the more difficult case by using nonlinear kernels. Recently [10] proposed to restrict the number of support vectors by solving the reduced support vector machines (RSVM) The main characteristic of this method is to reduce the matrix Q from l to l m, where m is the size of a randomly selected subset of training data considered as candidates of support ....

....smaller matrix can be stored in memory, so optimization algorithms such as Newton method can be applied. RSVM is different from directly solving smaller SVM problems with a subset of training because as the l constraints in the primal problem (1) are still kept during the optimization process. In [10] the authors showed the performance (testing accuracy) is as good as the regular SVM. As RSVM essentially uses less information than the regular SVM, we feel there is a need to conduct some serious comparisons. This is the first aim of the paper. Our experiments show that in general the testing ....

[Article contains additional citation context not shown here]

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. In Proceedings of the First SIAM International Conference on Data Mining, 2001.

....case to handle those outliers. Inspired by a geometric interpretation of the soft margin parameter D given by Bennett and Bredensteiner [BB00] we propose a method based on the use a random sampling technique. There is yet another recent alternative for SVM, the Reduced Support Vector Machine [LM01], for which it is proposed to use only single random subsample of data points, and to combine it to all the data points through kernel scalar products in feature space. The experimentation in [LM01] yields promising results but there is no theoretical guarantee either. Our approach has some ....

....technique. There is yet another recent alternative for SVM, the Reduced Support Vector Machine [LM01] for which it is proposed to use only single random subsample of data points, and to combine it to all the data points through kernel scalar products in feature space. The experimentation in [LM01] yields promising results but there is no theoretical guarantee either. Our approach has some resemblance with that approach in terms of the use of a random subset selection. However, in our algorithms, this is done repeatedly by filtering the selection through a probability distribution that ....

Y. Lee and O. L. Mangasarian, RSVM: Reduced Support Vector Machines, CD Proceedings of the SIAM International Conference on Data Mining, Chicago, April 5-7, 2001, SIAM, Philadelphia. (Available from http://www.cs.wisc.edu/~olvi/olvi.html.)

....the idea of solving approximations to RLSC. Hopefully, these approximations will yield a vast increase in speed with very little loss in accuracy. Several authors have suggested the use of low rank approximations to the kernel matrix in order to avoid explicit storage of the entire kernel matrix [19, 26, 20, 27, 22]. These techniques can be used in a variety of methods, including Regularized Least Squares Classification, Gaussian Process regression and classification, and interior point approaches to Support Vector Machines. The approaches all rely on choosing a subset of m of the training points (or a ....

Y.-J. Lee and O. Mangasarian, Rsvm: Reduced support vector machines, In SIAM International Conference on Data Mining (2001).

....case to handle those outliers. Inspired by a geometric interpretation of the soft margin parameter D given by Bennett and Bredensteiner [BB00] we propose a method based on the use a random sampling technique. There is yet another recent alternative for SVM, the Reduced Support Vector Machine [LM01], for which it is proposed to use only single random subsample of data points, and to combine it to all the data points through kernel scalar products in feature space. The experimentation in [LM01] yields promising results but there is no theoretical guarantee either. Our approach has some ....

....technique. There is yet another recent alternative for SVM, the Reduced Support Vector Machine [LM01] for which it is proposed to use only single random subsample of data points, and to combine it to all the data points through kernel scalar products in feature space. The experimentation in [LM01] yields promising results but there is no theoretical guarantee either. Our approach has some resemblance with that approach in terms of the use of a random subset selection. However, in our algorithms, this is done repeatedly by ltering the selection through a probability distribution that ....

Y. Lee and O. L. Mangasarian, RSVM: Reduced Support Vector Machines, CD Proceedings of the SIAM International Conference on Data Mining, Chicago, April 5-7, 2001, SIAM, Philadelphia. (Available from http://www.cs.wisc.edu/~olvi/olvi.html.) 23

....reduce the training time of SVM for large data sets are also based the same idea as the selective sampling which samples the data near the boundary with higher probabilities. They also need to scan the entire data set at each round when the samples are add in. Another method using random sampling [17] was developed for nonlinear SVM using the random sampling technique in the kernel trick. Based on the best of our knowledge, our proposed method is currently the only SVM for very large data sets which tries to generate the best results given limited amount of resource. 7. CONCLUSIONSAND ....

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. In First SIAM Int. Conf. Data Mining, Chicago, IL, 2001.

....feature space (or even an infinite dimensional space) which allows one to work implicitly with hyperplanes in such highly complex spaces. 3. 2 Notes on SVM kernels SVM polynomial kernel and Gaussian kernel have been widely used in pattern recognition due to its high expressible power [16, 3, 14]. Especially, Gaussian kernel has an infinite VC dimension, 1 which implies that it is basically able to draw any kinds of boundary func tions. e.g. In [14] Gaussian kernel draws a checkerboard shape of boundary. However, it is very hard to extract the feature weights from Gaussian kernel ....

....kernels SVM polynomial kernel and Gaussian kernel have been widely used in pattern recognition due to its high expressible power [16, 3, 14] Especially, Gaussian kernel has an infinite VC dimension, 1 which implies that it is basically able to draw any kinds of boundary func tions. e.g. In [14], Gaussian kernel draws a checkerboard shape of boundary. However, it is very hard to extract the feature weights from Gaussian kernel because it is very hard to express the kernel as an inner product of two (x) functions [7] Polynomial kernel on the other hand also has a higher expressive ....

Y.-J. Lee and O. L. Mangasarian. RSVM: Re- duced support vector machines. In First SIAM Int. Conf. Data Mining, Chicago, IL, 2001.

....for classi cation [17] The price paid for this extension is that problems with large datasets can be handled with the eciency of the linear case only if the inner product terms of the kernel [15, Equation (3) are explicitly known, which in general they are not. 8 Reduced kernel techniques [13, 25] may be bene cial here. Regardless, ASVR may be a useful tool for regression with nonlinear kernels because of its simplicity. ASVR does not require any specialized quadratic or linear programming code, but merely a linear equation solver which is publicly available. Future work should examine the ....

....on datasets with millions of points has been proposed and implemented. ASVR requires nothing more complex than a commonly available linear equation solver for solving small systems with few variables even for massive datasets. Future work includes implementing kernel reduction techniques [13, 25] to allow ASVR to be used eciently with nonlinear kernels. Acknowledgments Research described in this report was supported by a grant from the Howard Hughes Medical Institute, and by Carleton College. ....

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. In Proceedings of the First SIAM International Conference on Data Mining, Chicago, IL, April 2001.

....has been given on the efficiency of algorithms based on these techniques. As far as the authors know, the only positive theoretical results are the convergence (i.e. termination) of some of such algorithms [Lin01, KG01] Yet another recent alternative, the Reduced Support Vector Machine [LM01], proposes to use only a single random subsample of data points, and to combine it to all the data points through kernel computed scalar products in feature space. Our approach has some resemblance with this one in that a random selection of a small number of data points is made; however, in our ....

Y.-J. Lee and O. L. Mangasarian, RSVM: Reduced Support Vector Machines, in Proc. First SIAM International Conference on Data Mining, 2001.

....in the number of training data points. The rst approach is the subset of representers method (SRM) and can be found in the work of Wahba [5] in the work on sparse greedy Gaussian process regression by Smola and Bartlett [2] and in the reduced support vector machine by Lee and Mangasarian [1]. The SRM is based on a factorization of the kernel functions. The second variant is a reduced rank approximation (RRA) of the Gram matrix introduced in the work of Williams and Seeger [6] The RRA uses the same decomposition as the SRM but this decomposition is only applied to the Gram matrix. ....

Lee, Y.-J. and Mangasarian, O. L.: RSVM: Reduced Support Vector Machines. Data Mining Institute Technical Report 00-07, Computer Sciences Department, University of Wisconsin (2000)

....large sets of potential classifier points. In the next section, we consider a noisier version of this dataset and observe that classifiers constructed from small numbers of synthetic points actually yield better generalizability than those constructed from large numbers of training points. See [6] for analogous observations based on the use of small randomly 12 selected subsets of the training set as classifier points for a different collection of datasets. 4. Computational results, modified USPS problem In this section, we repeat part of the analysis above, on a (harder) modified ....

.... the resulting testing correctness of 75.47 is 15 actually worse than that obtained by using the classifier points from the much smaller sets C i (whose 100 points yield 76.76 classification) or Z i (whose 10 points yield 76.62 ) This apparently surprising result is analogous to results in [6], where random 1 5 subsets of the Adult Dataset [8] were allowed in the Gaussian kernel classifier. Observe that if the solution quality is measured by the number of support vectors, then by this measure as well, the smaller classifier sets provide better quality solutions than the full ....

[Article contains additional citation context not shown here]

Y.-J. Lee and O. L. Mangasarian, "RSVM: Reduced Support Vector Machines". Data Mining Institute technical report 00-07, July 2000.

....5000 4.9 85.86 2500 1.67 85.86 1000 0.41 84.81 500 0.19 83.08 100 0.02 69.31 Also, a block static implementation of ASVM was tested. For a given training set, a random block of b points was selected at the beginning, and all remaining training points were discarded. It has been shown [14] that this simple strategy produces a good testing accuracy in large sets. At each iteration, ASVM works with all constraints whose dual variables are nonzero. A brute force block dynamic ASVM implementation would simply pick a random sample of b constraints among them. Unfortunately, our tests ....

Y.-J. Lee and O.L. Mangasarian, RSVM: Reduced Support Vector Machines, Technical report 00-07, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, USA (July 2000).

....corresponding inequality constraint. Note further that the 2 norm of the error vector y is minimized instead of the 1 norm, and the margin between the bounding planes is maximized with respect to both orientation w and relative location to the origin . Extensive computational experience, as in [8, 9, 10, 6, 5] indicates that this formulation is just as good as the classical formulation (1) with some added advantages such as strong convexity of the objective function. The key idea of the proximal SVM is to make a very simple, but funda mental, change in the formulation (7) namely replace the ....

Y.-J. Lcc and O. L. Mangasarian. RSVM: Reduced support vector machines. Technical Report 00-07, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, July 2000.

....(15) We note that for most real world problems, the m x m kernel K(A, A ) is replaced by the considerably smaller m x m rectangular kernel K(A, A) where A consists of as little as 15 of the rows of A randomly chosen. This leads to an extremely fast and effective algorithm as described in [15]. Before presenting our numerical results we describe a simple computational refinement that consistently improved testing set correctness on all the datasets that we tested. 4 Newton Refinement The simple computational refinement that we have implemented, and which is applicable to any type of ....

....fast, in almost all the tested cases this refinement improved test set correctness of the MPSVM by as much as 10 . Nonlinear MPSVM: One From Rest classifier using a Nonlinear PSVM. A Gaussian kernel was used in all the computations. On the larger datasets (Vehicle, Segment) a rectangular kernel [15] was used in order to reduce even more the computational time while maintaining the correctness achieved by using the full kernel. The linear MPSVM running time was always at least two orders of magnitude less than the standard OFRQP time. Furthermore, there was no a significant statistical ....

[Article contains additional citation context not shown here]

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. Technical Report 00-07, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, July 2000.

....corresponding inequality constraint. Note further that the 2 norm of the error vector y is minimized instead of the 1 norm, and the margin between the bounding planes is maximized with respect to both orientation w and relative location to the origin if. Extensive computa tional experience, as in [22, 23, 24, 18, 17] indicates that this formulation is just as good as the classical formulation (2) with some added advantages such as strong convexity of the objective function. Our key idea in this present paper is to make a very simple, but very fundamental change in the formulation (8) namely replace the ....

.... (19) is given by (23) Unlike the situation with linear kernels, the Sherman Morrison Woodbury formula is useless here because the kernel matrix 14 K (A, A ) is a square m x m matrix, so the inversion must take place in a potentially high dimensional , However, the reduced kernel techniques of [17] can be utilized to reduce the m x m dimensionality of the kernel K K(A, A ) to a much smaller m x m dimensionality of a rectangular kernel K K(A,A ) where ra is as small as 1 of m and A is an rax n random submatrLx of of A. Such reduced kernels not only make most large problems tractable, but ....

[Article contains additional citation context not shown here]

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. Technical Report 00-07, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, July 2000.

No context found.

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. Technical Report 00-07, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, July 2000.

....Irvine datasets showing the percentage of reduction achieved over ten fold runs. Ten fold cross validation test correctness is the same for both the full kernel (mm) problem and the reduced kernel (m 1 m 2 ) because only non support vectors are discarded by the reduced kernel problem. RSVM [13] was used here in order to obtain a smaller initial kernel problem (8124 400 instead of 8124 8124) 7 Conclusion By utilizing a leave one out error bound, we have proposed an algorithm, based on solving a few linear programs, that generates an accurate nonlinear separating surface that is ....

Y.-J. Lee and O. L. Mangasarian. RSVM: Reduced support vector machines. Technical Report 00-07, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, July 2000. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/00-07.ps.

No context found.

Y. Lee and O. L. Mangasarian. RSVM: reduced support vector machines. In CD Proceedings of the First SIAM International Conference on Data Mining, Chicago, 2001.

No context found.

Y.-J. Lee and O. Mangasarian. RSVM: Reduced support vector machines. In First SIAM International Conference on Data Mining, Chicago, 2001.

No context found.

Y-J. Lee and O.L. Mangasarian. RSVM: Reduced Support Vector Machines. In CD Proceedings of the SIAM International Conference on Data Mining, Chicago, April 5-7, 2001.

No context found.

Y-J. Lee and O.L. Mangasarian. RSVM: reduced support vector machines. In Proc. SIAM Intl. Conf. on Data Mining, Chicago, 2001.

No context found.

Y-J. Lee and O.L. Mangasarian. RSVM: reduced support vector machines. In Proc. SIAM Intl. Conf. on Data Mining, Chicago, 2001.

No context found.

Y. Lee and O. L. Mangasarian, "RSVM: Reduced support vector machines, " First SIAM International Conference on Data Mining, Chicago, April 2001.

No context found.

Y. Lee, O. L. Mangasarian, RSVM: Reduced support vector machines, First SIAM International Conference on Data Mining, Chicago, technical Report 0007, July, 2000.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC