In this paper, we derive a new subspace algorithm to consistently identify stochastic state space models from given output data without forming the covariance matrix and using only semi-infinite block Hankel matrices. The algorithm is based on the concept of principal angles and directions. We describe how they can be calculated with QR and Quotient Singular Value Decomposition. We also provide an interpretation of the principal directions as states of a non-steady state Kalman filter bank. Key Words Principal angles and directions, QR and quotient singular value decomposition, Kalman filter, Riccati difference equation, stochastic balancing, stochastic realization. 1 Introduction Let y k 2 ! l ; k = 0; 1; : : : ; K be a data sequence that is generated by the following system : x k+1 = Ax k + w k (1) y k = Cx k + v k (2) where x k 2 ! n is a state vector. The vector sequence w k 2 ! n is a process noise while the vector sequence v k 2 ! l is a measurement noise. They are bo...

This paper surveys the contributions of five mathematicians --- Eugenio Beltrami (1835--1899), Camille Jordan (1838--1921), James Joseph Sylvester (1814--1897), Erhard Schmidt (1876--1959), and Hermann Weyl (1885--1955) --- who were responsible for establishing the existence of the singular value decomposition and developing its theory.

. It is shown that the basic regularization procedures for finding meaningful approximate solutions of ill-conditioned or singular linear systems can be phrased and analyzed in terms of classical linear algebra that can be taught in any numerical analysis course. Apart from rewriting many known results in a more elegant form, we also derive a new two-parameter family of merit functions for the determination of the regularization parameter. The traditional merit functions from generalized cross validation (GCV) and generalized maximum likelihood (GML) are recovered as special cases. Key words. regularization, ill-posed, ill-conditioned, generalized cross validation, generalized maximum likelihood, Tikhonov regularization, error bounds AMS subject classifications. primary 65F05; secondary 65J20 1. Introduction. In many applications of linear algebra, the need arises to find a good approximation x to a vector x 2 IR n satisfying an approximate equation Ax ß y with ill-conditioned o...

Discriminant analysis has been used for decades to extract features that preserve class separability. It is commonly defined as an optimization problem involving covariance matrices that represent the scatter within and between clusters. The requirement that one of these matrices be nonsingular limits its application to data sets with certain relative dimensions. We examine a number of optimization criteria, and extend their applicability by using the generalized singular value decomposition to circumvent the nonsingularity requirement. The result is a generalization of discriminant analysis that can be applied even when the sample size is smaller than the dimension of the sample data. We use classification results from the reduced representation to compare the effectiveness of this approach with some alternatives, and conclude with a discussion of their relative merits. 1

Abstract. In today’s vector space information retrieval systems, dimension reduction is imperative for efficiently manipulating the massive quantity of data. To be useful, this lower-dimensional representation must be a good approximation of the full document set. To that end, we adapt and extend the discriminant analysis projection used in pattern recognition. This projection preserves cluster structure by maximizing the scatter between clusters while minimizing the scatter within clusters. A common limitation of trace optimization in discriminant analysis is that one of the scatter matrices must be nonsingular, which restricts its application to document sets in which the number of terms does not exceed the number of documents. We show that by using the generalized singular value decomposition (GSVD), we can achieve the same goal regardless of the relative dimensions of the term-document matrix. In addition, applying the GSVD allows us to avoid the explicit formation of the scatter matrices in favor of working directly with the data matrix, thus improving the numerical properties of the approach. Finally, we present experimental results that confirm the effectiveness of our approach.

Abstract—An optimization criterion is presented for discriminant analysis. The criterion extends the optimization criteria of the classical Linear Discriminant Analysis (LDA) through the use of the pseudoinverse when the scatter matrices are singular. It is applicable regardless of the relative sizes of the data dimension and sample size, overcoming a limitation of classical LDA. The optimization problem can be solved analytically by applying the Generalized Singular Value Decomposition (GSVD) technique. The pseudoinverse has been suggested and used for undersampled problems in the past, where the data dimension exceeds the number of data points. The criterion proposed in this paper provides a theoretical justification for this procedure. An approximation algorithm for the GSVD-based approach is also presented. It reduces the computational complexity by finding subclusters of each cluster and uses their centroids to capture the structure of each cluster. This reduced problem yields much smaller matrices to which the GSVD can be applied efficiently. Experiments on text data, with up to 7,000 dimensions, show that the approximation algorithm produces results that are close to those produced by the exact algorithm. Index Terms—Classification, clustering, dimension reduction, generalized singular value decomposition, linear discriminant analysis, text mining. 1

Abstract not found

We present a variation of Paige's algorithm for computing the generalized singular value decomposition (GSVD) of two matrices A and B. There are two innovations. The first is a new preprocessing step which reduces A and B to upper triangular forms satisfying certain rank conditions. The second is a new 2 \Theta 2 triangular GSVD algorithm, which constitutes the inner loop of Paige's algorithm. We present proofs of stability and high accuracy of the 2 \Theta 2 GSVD algorithm, and demonstrate it using examples on which all previous algorithms fail. 1 Introduction The purpose of this paper is to describe a variation of Paige's algorithm [28] for computing the following generalized singular value decomposition (GSVD) introduced by Van Loan [33], and Paige and Saunders [25]. This is also called the quotient singular value decomposition (QSVD) in [8]. Theorem 1.1 Let A 2 IR m\Thetan and B 2 IR p\Thetan have rank(A T ; B T ) = n. 1 Then there are orthogonal matrices U , V and Q su...

Support vector machines (SVMs) have been recognized as one of the most successful classification methods for many applications including text classification. Even though the learning ability and computational complexity of training in support vector machines may be independent of the dimension of the feature space, reducing computational complexity is an essential issue to efficiently handle a large number of terms in practical applications of text classification. In this paper, we adopt novel dimension reduction methods to reduce the dimension of the document vectors dramatically. We also introduce decision functions for the centroid-based classification algorithm and support vector classifiers to handle the classification problem where a document may belong to multiple classes. Our substantial experimental results show that with several dimension reduction methods that are designed particularly for clustered data, higher efficiency for both training and testing can be achieved without sacrificing prediction accuracy of text classification even when the dimension of the input space is significantly reduced.

. In this paper, we discuss multi-matrix generalizations of two well-known orthogonal rank factorizations of a matrix: the generalized singular value decomposition and the generalized QR-(or URV-) decomposition. These generalizations can be obtained for any number of matrices of compatible dimensions. We discuss in detail the structure of these generalizations and their mutual relations and give a constructive proof for the generalized QR-decompositions. Key words. singular value decomposition, QR-factorization, URV-decomposition, complete orthogonal decomposition 1. Introduction. In this paper, we present multi-matrix generalizations of some well-known orthogonal rank factorizations. We will show how the idea of a QR- decomposition (QRD), a URV-decomposition (URVD) and a singular value decomposition (SVD) for one matrix can be generalized to any number of matrices. While generalizations of the SVD for any number of matrices have been derived in [9], one of the main contributions ...