Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

...ent of the th feature in the observed data vector . The use of ICA for feature extraction is motivated by results in neurosciences that suggest that the similar principle of redundancy reduction [2], =-=[32]-=- explains some aspects of the early processing of sensory data by the brain. ICA has also applications in exploratory data analysis in the same way as the closely related method of projection pursuit ...

Citation Context

..., we shall derive the fixed-point algorithm for one unit, with sphered data. First note that the maxima of JG(w) are obtained at certain optima of E{G(w T x)}. According to the Kuhn-Tucker conditions =-=[29]-=-, the optima of E{G(w T x)} under the constraint E{(w T x) 2 } = �w� 2 = 1 are obtained at points where E{xg(w T x)} − βw = 0 (17) where β is a constant that can be easily evaluated to give β = E{w T ...

Citation Context

...ltiplying both sides of the first equation in (19) by β − E{g ′ (w T x)}. This gives the following fixed-point algorithm w + = E{xg(w T x)} − E{g ′ (w T x)}w w ∗ = w + /�w + � which was introduced in =-=[17]-=- using a more heuristic derivation. An earlier version (for kurtosis only) was derived as a fixed-point iteration of a neural learning rule in [23], which is where its name comes from. We retain this ...

Citation Context

...each other, and thus the signals can be recovered from linear mixtures xi by finding a transformation in which the transformed signals are as independent as possible, as in ICA. In feature extraction =-=[4, 25]-=-, si is the coefficient of the i-th feature in the observed data vector x. The use of ICA for feature extraction is motivated by results in neurosciences that suggest that the similar principle of red...

Citation Context

...es and methods have been developed to find such a linear representation, including principal component analysis [30], factor analysis [15], projection pursuit [12, 16], independent component analysis =-=[27]-=-, etc. The transformation may be defined using such criteria as optimal dimension reduction, statistical ’interestingness’ of the resulting components si, simplicity of the transformation, or other cr...

Citation Context

...ariables s = Wx (1) has some suitable properties. Several principles and methods have been developed to find such a linear representation, including principal component analysis [30], factor analysis =-=[15]-=-, projection pursuit [12, 16], independent component analysis [27], etc. The transformation may be defined using such criteria as optimal dimension reduction, statistical ’interestingness’ of the resu...

Citation Context

...nsformation is to find a representation in which the transformed components si are statistically as independent from each other as possible. Thus this method is a special case of redundancy reduction =-=[2]-=-. Two promising applications of ICA are blind source separation and feature extraction. In blind source separation [27], the observed values of x correspond to a realization of an m-dimensional discre...

Citation Context

...t functions and algorithms introduced above. These applications include artifact cancellation in EEG and MEG [36, 37], decomposition of evoked fields in MEG [38], and feature extraction of image data =-=[35, 25]-=-. These experiments further validate the ICA methods introduced in this paper. A Matlab T M implementation of the fixed-algorithm is available on the World Wide Web free of charge [10]. 10slog10 of es...

Citation Context

...he mutual information is roughly equivalent to finding directions in which the negentropy is maximized. This formulation of ICA also shows explicitly the connection between ICA and projection pursuit =-=[11, 12, 16, 26]-=-. In fact, finding a single direction that maximizes negentropy is a form of projection pursuit, and could also be interpreted as estimation of a single independent component [24]. 2.2 Contrast Functi...

Citation Context

...ome suitable properties. Several principles and methods have been developed to find such a linear representation, including principal component analysis [30], factor analysis [15], projection pursuit =-=[12, 16]-=-, independent component analysis [27], etc. The transformation may be defined using such criteria as optimal dimension reduction, statistical ’interestingness’ of the resulting components si, simplici...

Citation Context

...ome suitable properties. Several principles and methods have been developed to find such a linear representation, including principal component analysis [30], factor analysis [15], projection pursuit =-=[12, 16]-=-, independent component analysis [27], etc. The transformation may be defined using such criteria as optimal dimension reduction, statistical ’interestingness’ of the resulting components si, simplici...

Citation Context

...and distributions of the si. In particular, if G(u) = u 4 , the condition is fulfilled for any distribution of non-zero kurtosis. In that case, it can also be proven that there are no spurious optima =-=[9]-=-. 3.1.2 Asymptotic variance Asymptotic variance is one criterion for choosing the function G to be used in the contrast function. Comparison of, say, the traces of the asymptotic covariance matrices o...

Citation Context

... the transformation, or other criteria, including application-oriented ones. We treat in this paper the problem of estimating the transformation given by (linear) independent component analysis (ICA) =-=[7, 27]-=-. As the name implies, the basic goal in determining the transformation is to find a representation in which the transformed components si are statistically as independent from each other as possible....

Citation Context

...tion of the observed variables s = Wx (1) has some suitable properties. Several principles and methods have been developed to find such a linear representation, including principal component analysis =-=[30]-=-, factor analysis [15], projection pursuit [12, 16], independent component analysis [27], etc. The transformation may be defined using such criteria as optimal dimension reduction, statistical ’intere...

Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

... of si, and k1,k2,k3 are arbitrary constants. For simplicity, one can choose Gopt(u) = log fi(u). Thus the optimal contrast function is the same as the one obtained by the maximum likelihood approach =-=[34]-=-, or the infomax approach [3]. Almost identical results have also been obtained in [5] for another algorithm. The theorem above treats, however, the one-unit case instead of the multi-unit case treate...

Citation Context

...treats, however, the one-unit case instead of the multi-unit case treated by the other authors. 4 (10)s3.1.3 Robustness Another very attractive property of an estimator is robustness against outliers =-=[14]-=-. This means that single, highly erroneous observations do not have much influence on the estimator. To obtain a simple form of robustness called B-robustness, we would like the estimator to have a bo...

Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

...he mutual information is roughly equivalent to finding directions in which the negentropy is maximized. This formulation of ICA also shows explicitly the connection between ICA and projection pursuit =-=[11, 12, 16, 26]-=-. In fact, finding a single direction that maximizes negentropy is a form of projection pursuit, and could also be interpreted as estimation of a single independent component [24]. 2.2 Contrast Functi...

Citation Context

...cient of the i-th feature in the observed data vector x. The use of ICA for feature extraction is motivated by results in neurosciences that suggest that the similar principle of redundancy reduction =-=[2, 32]-=- explains some aspects of the early processing of sensory data by the brain. ICA has also applications in exploratory data analysis in the same way as the closely related method of projection pursuit ...

Citation Context

...ough Approximations of Negentropy To use the definition of ICA given above, a simple estimate of the negentropy (or of differential entropy) is needed. We use here the new approximations developed in =-=[19]-=-, based on the maximum entropy principle. In [19] it was shown that these approximations are often considerably more accurate than the conventional, cumulant-based approximations in [7, 1, 26]. In the...

Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

...Experiments on different kinds of real life data have also been performed using the contrast functions and algorithms introduced above. These applications include artifact cancellation in EEG and MEG =-=[36, 37]-=-, decomposition of evoked fields in MEG [38], and feature extraction of image data [35, 25]. These experiments further validate the ICA methods introduced in this paper. A Matlab T M implementation of...

Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

...Experiments on different kinds of real life data have also been performed using the contrast functions and algorithms introduced above. These applications include artifact cancellation in EEG and MEG =-=[36, 37]-=-, decomposition of evoked fields in MEG [38], and feature extraction of image data [35, 25]. These experiments further validate the ICA methods introduced in this paper. A Matlab T M implementation of...

Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

...neither sphering nor inversion of the covariance matrix is needed. In fact, such an algorithm can be considered as a fixed-point algorithm for maximum likelihood estimation of the ICA data model, see =-=[21]-=-. 4.4 Properties of the Fixed-Point Algorithm The fixed-point algorithm and the underlying contrast functions have a number of desirable properties when compared with existing methods for ICA. • The c...

Citation Context

... G to be used in the contrast function. Comparison of, say, the traces of the asymptotic covariance matrices of two estimators enables direct comparison of the mean-square error of the estimators. In =-=[18]-=-, evaluation of asymptotic variances was addressed using a related family of contrast functions. In fact, it can be seen that the results in [18] are valid even in this case, and thus we have the foll...

Citation Context

... observed variables has some suitable properties. Several principles and methods have been developed to find such a linear representation, including principal component analysis [30], factor analysis =-=[15]-=-, projection pursuit [12], [16], independent component analysis [27], etc. The transformation may be defined using such criteria as optimal dimension reduction, statistical “interestingness” of the re...

Citation Context

...alidated the novel contrast functions and algorithms introduced. Some extensions of the methods introduced in this paper are presented in [20], in which the problem of noisy data is addressed, and in =-=[22]-=-, which deals with the situation where there are more independent components than observed variables. APPENDIX A PROOFS A. Proof of Convergence of Algorithm (20) The convergence is proven under the as...

Citation Context

...t functions and algorithms introduced above. These applications include artifact cancellation in EEG and MEG [36, 37], decomposition of evoked fields in MEG [38], and feature extraction of image data =-=[35, 25]-=-. These experiments further validate the ICA methods introduced in this paper. A Matlab T M implementation of the fixed-algorithm is available on the World Wide Web free of charge [10]. 10slog10 of es...

Citation Context

... to achieve the maximum accuracy allowed by the data. This illustrates the fast convergence of the fixed-point algorithm. In fact, a comparison of our algorithm with other algorithms was performed in =-=[13]-=-, showing that the fixed-point algorithm gives approximately the same statistical efficiency as other algorithms, but with a fraction of the computational cost. Experiments on different kinds of real ...

Citation Context

... have also been performed using the contrast functions and algorithms introduced above. These applications include artifact cancellation in EEG and MEG [36, 37], decomposition of evoked fields in MEG =-=[38]-=-, and feature extraction of image data [35, 25]. These experiments further validate the ICA methods introduced in this paper. A Matlab T M implementation of the fixed-algorithm is available on the Wor...

Citation Context

...ough Approximations of Negentropy To use the definition of ICA given above, a simple estimate of the negentropy (or of differential entropy) is needed. We use here the new approximations developed in =-=[19]-=-, based on the maximum entropy principle. In [19] it was shown that these approximations are often considerably more accurate than the conventional, cumulant-based approximations in [1], [7], and [26]...

Citation Context

... ICA data model, minimization of mutual information, and projection pursuit One popular way of formulating the ICA problem is to consider the estimation of the following generative model for the data =-=[1, 3, 5, 6, 23, 24, 27, 28, 31]-=-: x = As (2) where x is an observed m-dimensional vector, s is an n-dimensional (latent) random vector whose components are assumed mutually independent, and A is a constant m × n matrix to be estimat...

Citation Context

...on to be used in the contrast function. Comparison of, say, the traces of the asymptotic covariance matrices of two estimators enables direct comparison of the mean-square error of the estimators. In =-=[18]-=-, evaluation of asymptotic variances was addressed using a related family of contrast functions. In fact, it can be seen that the results in [18] are valid even in this case, and thus we have the foll...

Citation Context

.... Simulations as well as applications on real-life data have validated the novel contrast functions and algorithms introduced. Some extensions of the methods introduced in this paper are presented in =-=[20]-=-, in which the problem of noisy data is addressed, and in [22], which deals with the situation where there are more independent components than observed variables. APPENDIX A PROOFS A. Proof of Conver...

Citation Context

...e limited to: Helsingin Yliopisto. Downloaded on March 23, 2009 at 08:38 from IEEE Xplore. Restrictions apply.HYVÄRINEN: FAST AND ROBUST FIXED-POINT ALGORITHMS 627 the data [1], [3], [5], [6], [23], =-=[24]-=-, [27], [28], [31]: where is an observed -dimensional vector, is an - dimensional (latent) random vector whose components are assumed mutually independent, and is a constant matrix to be estimated. It...

Citation Context

...ingin Yliopisto. Downloaded on March 23, 2009 at 08:38 from IEEE Xplore. Restrictions apply.HYVÄRINEN: FAST AND ROBUST FIXED-POINT ALGORITHMS 627 the data [1], [3], [5], [6], [23], [24], [27], [28], =-=[31]-=-: where is an observed -dimensional vector, is an - dimensional (latent) random vector whose components are assumed mutually independent, and is a constant matrix to be estimated. It is usually furthe...