Fast and Robust Fixed-Point Algorithms for Independent Component Analysis
Fast and Robust Fixed-Point Algorithms for Independent Component Analysis
Aapo Hyvärinen Helsinki University of Technology Laboratory of Computer and Information Science P.O. Box 5400, FIN-02015 HUT, Finland Email: aapo.hyvarinen@@hut.fi IEEE Trans. on Neural Networks, 10(3):626-634, 1999.
Abstract Independent component analysis (ICA) is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible. In this paper, we use a combination of two different approaches for linear ICA: Comon’s information-theoretic approach and the projection pursuit approach. Using maximum entropy approximations of differential entropy, we introduce a family of new contrast (objective) functions for ICA. These contrast functions enable both the estimation of the whole decomposition by minimizing mutual information, and estimation of individual independent components as projection pursuit directions. The statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model, and it is shown how to choose contrast functions that are robust and/or of minimum variance. Finally, we introduce simple fixed-point algorithms for practical optimization of the contrast functions. These algorithms optimize the contrast functions very fast and reliably.
1
Introduction
A central problem in neural network research, as well as in statistics and signal processing, is finding a suitable representation or transformation of the data. For computational and conceptual simplicity, the representation is often sought as a linear transformation of the original data. Let us denote by x = (x1 , x2 , ..., xm )T a zero-mean m-dimensional random variable that can be observed, and by s = (s1 , s2 , ..., sn )T its n-dimensional transform. Then the problem is to determine a constant (weight) matrix W so that the linear transformation of the observed
Aapo Hyvärinen Helsinki University of Technology Laboratory of Computer and Information Science P.O. Box 5400, FIN-02015 HUT, Finland Email: aapo.hyvarinen@@hut.fi IEEE Trans. on Neural Networks, 10(3):626-634, 1999.
Abstract Independent component analysis (ICA) is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible. In this paper, we use a combination of two different approaches for linear ICA: Comon’s information-theoretic approach and the projection pursuit approach. Using maximum entropy approximations of differential entropy, we introduce a family of new contrast (objective) functions for ICA. These contrast functions enable both the estimation of the whole decomposition by minimizing mutual information, and estimation of individual independent components as projection pursuit directions. The statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model, and it is shown how to choose contrast functions that are robust and/or of minimum variance. Finally, we introduce simple fixed-point algorithms for practical optimization of the contrast functions. These algorithms optimize the contrast functions very fast and reliably.
1
Introduction
A central problem in neural network research, as well as in statistics and signal processing, is finding a suitable representation or transformation of the data. For computational and conceptual simplicity, the representation is often sought as a linear transformation of the original data. Let us denote by x = (x1 , x2 , ..., xm )T a zero-mean m-dimensional random variable that can be observed, and by s = (s1 , s2 , ..., sn )T its n-dimensional transform. Then the problem is to determine a constant (weight) matrix W so that the linear transformation of the observed