by Fabian J. Theis, Elmar W. Lang
in Proc. Int. Conf. on Independent Component Analysis and Signal Separation (ICA2001
http://homepages.uni-regensburg.de/~thf11669/publications/theis01memmi_ICA01.pdf
Add To MetaCart
Abstract:
In blind source separation, two different separation techniques are mainly used: Minimal Mutual Information (MMI), where minimization of the mutual output information yields an independent random vector, and Maximum Entropy (ME), where the output entropy is maximized. However, it is yet unclear why ME should solve the separation problem, ie. result in an independent vector. Amari has given a partial confirmation for ME in the linear case in [1], where he proves that under the assumption of vanishing expectancy of the sources ME does not change the solutions of MMI up to scaling and permutation. In this paper, we generalize Amari’s approach to nonlinear ICA problems, where random vectors have been mixed by output functions of layered neural networks. We show that certain solution points of MMI are kept fixed by ME if no scaling of the weight vectors is allowed. In general, ME however might leave those MMI solutions using diagonal weights in the first network layer. Therefore, we conclude this paper by suggesting that in nonlinear ME algorithms diagonal weights should be fixed in later epochs. 1.
Citations
|
808
|
Independent component analysis, a new concept
– Comon
- 1994
|
|
800
|
Multilayer feedforward networks are universal approximators
– Hornik, Stinchcombe, et al.
- 1989
|
|
688
|
An information-maximization approach to blind separation and blind deconvolution
– Bell, Sejnowski
- 1995
|
|
618
|
Survey on independent component analysis
– Hyvärinen
- 1999
|
|
345
|
Blind separation of sources, Part I: an adaptive algorithm based on a neuromimetic architecture
– Jutten, Hérault
- 1991
|
|
194
|
Neural Network Learning: Theoretical Foundations
– ANTHONY, BARTLETT
- 1999
|
|
193
|
Neural Networks
– Haykin
- 1993
|
|
122
|
Nonlinear neuron in the low noise limit: A factorial code maximizes information transfer
– Nadal, Parga
- 1994
|
|
98
|
Adaptive online learning algorithms for blind separation: maximum entropy and minimum mutual information
– Yang, Amari
- 1997
|
|
82
|
Source separation in post-nonlinear mixtures
– Taleb, Jutten
- 1999
|
|
79
|
Independent component analysis: theory and applications
– Lee
- 1998
|
|
49
|
An application of the principle of maximum information preservation to linear systems
– Linsker
- 1989
|
|
45
|
Local synaptic learning rules suffice to maximize mutual information in a linear network
– Linsker
- 1992
|
|
36
|
Information-theoretic approach to blind separation of sources
– Yang, Amari, et al.
- 1998
|
|
23
|
Separation of nonlinear mixtures using pattern repulsion
– Marques, Almeida
- 1999
|
|
9
|
Blind separation of nonlinear mixing models
– Lee, Koehler, et al.
- 1997
|
|
5
|
On existence and uniqueness of solutions in nonlinear independent component analysis
– Hyvärinen, Pajunen
- 1998
|
|
4
|
Nonlinear approaches to independent component analysis
– Lee
- 1999
|
