A "nonnegative PCA" algorithm for independent component analysis

We consider the task of independent component analysis when the independent sources are known to be nonnegative and well-grounded, so that they have a nonzero probability density function (pdf) in the region of zero. We propose the use of a "nonnegative principal component analysis (nonnegative PCA)" algorithm, which is a special case of the nonlinear PCA algorithm, but with a rectification nonlinearity, and we conjecture that this algorithm will find such nonnegative well-grounded independent sources, under reasonable initial conditions. While the algorithm has proved difficult to analyze in the general case, we give some analytical results that are consistent with this conjecture and some numerical simulations that illustrate its operation.     

Maximum contrast analysis for nonnegative blind source separation

In this paper, we propose a maximum contrast analysis (MCA) method for nonnegative blind source separation, where both the mixing matrix and the source signals are nonnegative. We first show that the contrast degree of the source signals is greater than that of the mixed signals. Motivated by this observation, we propose an MCA-based cost function. It is further shown that the separation matrix can be obtained by maximizing the proposed cost function. Then we derive an iterative determinant maximization algorithm for estimating the separation matrix. In the case of two sources, a closed-form solution exists and is derived. Unlike most existing blind source separation methods, the proposed MCA method needs neither the independence assumption, nor the sparseness requirement of the sources. The effectiveness of the new method is illustrated by experiments using X-ray images, remote sensing images, infrared spectral images, and real-world fluorescence microscopy images.     

On the Effect of the Form of the Posterior Approximation in Variational Learning of ICA Models

We show that the choice of posterior approximation affects the solution found in Bayesian variational learning of linear independent component analysis models. Assuming the sources to be independent a posteriori favours a solution which has orthogonal mixing vectors. Linear mixing models with either temporally correlated sources or non-Gaussian source models are considered but the analysis extends to nonlinear mixtures as well.     

A null space method for over-complete blind source separation

In blind source separation, there are M sources that produce sounds independently and continuously over time. These sounds are then recorded by m receivers. The sound recorded by each receiver at each time point is a linear superposition of the sounds produced by the M sources at the same time point. The problem of blind source separation is to recover the sounds of the sources from the sounds recorded by the receivers, without knowledge of the m×M mixing matrix that transforms the sounds of the sources to the sounds of the receivers at each time point. Over-complete separation refers to the situation where the number of sources M is greater than the number of receivers m, so that the source sounds cannot be uniquely solved from the receiver sounds even if the mixing matrix is known. In this paper, we propose a null space representation for the over-complete blind source separation problem. This representation explicitly identifies the solution space of the source sounds in terms of the null space of the mixing matrix using singular value decomposition. Under this representation, the problem can be posed in the framework of Bayesian latent variable model, where the mixing matrix and the source sounds can be inferred based on their posterior distributions. We then propose a null space algorithm for Markov chain Monte Carlo posterior sampling. We illustrate the algorithm using several examples under two different statistical assumptions about the independent source sounds. The blind source separation problem is mathematically equivalent to the independent component analysis problem. So our method can be equally applied to over-complete independent component analysis for unsupervised learning of high-dimensional data.     

Independent factor analysis

We introduce the independent factor analysis (IFA) method for recovering independent hidden sources from their observed mixtures. IFA generalizes and unifies ordinary factor analysis (FA), principal component analysis (PCA), and independent component analysis (ICA), and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy. IFA is a two-step procedure. In the first step, the source densities, mixing matrix, and noise covariance are estimated from the observed data by maximum likelihood. For this purpose we present an expectation-maximization (EM) algorithm, which performs unsupervised learning of an associated probabilistic model of the mixing situation. Each source in our model is described by a mixture of gaussians; thus, all the probabilistic calculations can be performed analytically. In the second step, the sources are reconstructed from the observed data by an optimal nonlinear estimator. A variational approximation of this algorithm is derived for cases with a large number of sources, where the exact algorithm becomes intractable. Our IFA algorithm reduces to the one for ordinary FA when the sources become gaussian, and to an EM algorithm for PCA in the zero-noise limit. We derive an additional EM algorithm specifically for noiseless IFA. This algorithm is shown to be superior to ICA since it can learn arbitrary source densities from the data. Beyond blind separation, IFA can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding.     

Global convergence analysis of a discrete time nonnegative ICA algorithm

When the independent sources are known to be nonnegative and well-grounded, which means that they have a nonzero pdf in the region of zero, Oja and Plumbley have proposed a "Nonnegative principal component analysis (PCA)" algorithm to separate these positive sources. Generally, it is very difficult to prove the convergence of a discrete-time independent component analysis (ICA) learning algorithm. However, by using the skew-symmetry property of this discrete-time "Nonnegative PCA" algorithm, if the learning rate satisfies suitable condition, the global convergence of this discrete-time algorithm can be proven. Simulation results are employed to further illustrate the advantages of this theory.     

Mean-field approaches to independent component analysis

We develop mean-field approaches for probabilistic independent component analysis (ICA). The sources are estimated from the mean of their posterior distribution and the mixing matrix (and noise level) is estimated by maximum a posteriori (MAP). The latter requires the computation of (a good approximation to) the correlations between sources. For this purpose, we investigate three increasingly advanced mean-field methods: the variational (also known as naive mean field) approach, linear response corrections, and an adaptive version of the Thouless, Anderson and Palmer (1977) (TAP) mean-field approach, which is due to Opper and Winther (2001). The resulting algorithms are tested on a number of problems. On synthetic data, the advanced mean-field approaches are able to recover the correct mixing matrix in cases where the variational meanfield theory fails. For handwritten digits, sparse encoding is achieved using nonnegative source and mixing priors. For speech, the mean-field method is able to separate in the underdetermined (overcomplete) case of two sensors and three sources. One major advantage of the proposed method is its generality and algorithmic simplicity. Finally, we point out several possible extensions of the approaches developed here.     

Monotonic convergence of fixed-point algorithms for ICA

We re-examine a fixed-point algorithm proposed by Hyvarinen for independent component analysis, wherein local convergence is proved subject to an ideal signal model using a square invertible mixing matrix. Here, we derive step-size bounds which ensure monotonic convergence to a local extremum for any initial condition. Our analysis does not assume an ideal signal model but appeals rather to properties of the contrast function itself, and so applies even with noisy data and/or more sources than sensors. The results help alleviate the guesswork that often surrounds step-size selection when the observed signal does not fit an idealized model.     

A dynamical systems approach to weighted graph matching

Graph matching is a fundamental problem that arises frequently in the areas of distributed control, computer vision, and facility allocation. In this paper, we consider the optimal graph matching problem for weighted graphs, which is computationally challenging due the combinatorial nature of the set of permutations. Contrary to optimization-based relaxations to this problem, in this paper we develop a novel relaxation by constructing dynamical systems on the manifold of orthogonal matrices. In particular, since permutation matrices are orthogonal matrices with nonnegative elements, we define two gradient flows in the space of orthogonal matrices. The first minimizes the cost of weighted graph matching over orthogonal matrices, whereas the second minimizes the distance of an orthogonal matrix from the finite set of all permutations. The combination of the two dynamical systems converges to a permutation matrix, which provides a suboptimal solution to the weighted graph matching problem. Finally, our approach is shown to be promising by illustrating it on nontrivial problems.     

Algorithms for nonnegative independent component analysis

We consider the task of solving the independent component analysis (ICA) problem x=As given observations x, with a constraint of nonnegativity of the source random vector s. We refer to this as nonnegative independent component analysis and we consider methods for solving this task. For independent sources with nonzero probability density function (pdf) p(s) down to s=0 it is sufficient to find the orthonormal rotation y=Wz of prewhitened sources z=Vx, which minimizes the mean squared error of the reconstruction of z from the rectified version y/sup +/ of y. We suggest some algorithms which perform this, both based on a nonlinear principal component analysis (PCA) approach and on a geodesic search method driven by differential geometry considerations. We demonstrate the operation of these algorithms on an image separation problem, which shows in particular the fast convergence of the rotation and geodesic methods and apply the approach to a musical audio analysis task.     

Adaptive improved natural gradient algorithm for blind source separation

We propose an adaptive improved natural gradient algorithm for blind separation of independent sources. First, inspired by the well-known backpropagation algorithm, we incorporate a momentum term into the natural gradient learning process to accelerate the convergence rate and improve the stability. Then an estimation function for the adaptation of the separation model is obtained to adaptively control a step-size parameter and a momentum factor. The proposed natural gradient algorithm with variable step-size parameter and variable momentum factor is therefore particularly well suited to blind source separation in a time-varying environment, such as an abruptly changing mixing matrix or signal power. The expected improvement in the convergence speed, stability, and tracking ability of the proposed algorithm is demonstrated by extensive simulation results in both time-invariant and time-varying environments. The ability of the proposed algorithm to separate extremely weak or badly scaled sources is also verified. In addition, simulation results show that the proposed algorithm is suitable for separating mixtures of many sources (e.g., the number of sources is 10) in the complete case.     

A class of complex ICA algorithms based on the kurtosis cost function.

In this paper, we introduce a novel way of performing real-valued optimization in the complex domain. This framework enables a direct complex optimization technique when the cost function satisfies the Brandwood's independent analyticity condition. In particular, this technique has been used to derive three algorithms, namely, kurtosis maximization using gradient update (KM-G), kurtosis maximization using fixed-point update (KM-F), and kurtosis maximization using Newton update (KM-N), to perform the complex independent component analysis (ICA) based on the maximization of the complex kurtosis cost function. The derivation and related analysis of the three algorithms are performed in the complex domain without using any complex-real mapping for differentiation and optimization. A general complex Newton rule is also derived for developing the KM-N algorithm. The real conjugate gradient algorithm is extended to the complex domain similar to the derivation of complex Newton rule. The simulation results indicate that the fixed-point version (KM-F) and gradient version (KM-G) are superior to other similar algorithms when the sources include both circular and noncircular distributions and the dimension is relatively high.     

基于最大信息理论和共轭梯度寻优的ICA算法

独立分量分析是一种将观测向量分解为若干个独立统计的分量的一种统计学方法。提出了一种新的独立分量分析方法,该方法在最大信息理论的基础上引入目标函数,并利用共轭梯度搜索算法替代自然梯度算法,推导出用于训练转换矩阵的学习方程。运用核密度函数估算方法自适应地估算学习方程中包含的评价函数项。仿真结果表明,提出的基于独立分量分析的共轭梯度算法在求解盲源分离问题中切实有效。     

基于约束NMF的欠定盲信号分离算法*

提出一种约束非负矩阵分解方法用于解决欠定盲信号分离问题。非负矩阵分解直接用于求解欠定盲信号分离时,分解结果不唯一,无法正确分离源信号。本文在基本非负矩阵分解算法基础上,对分解得到的混合矩阵施加行列式约束,保证分解结果的唯一性;对分解得到的源信号同时施加稀疏性约束和最小相关约束,实现混合信号的唯一分解,提高源信号分离性能。仿真实验证明了本文算法的有效性。     

Nonnegative matrix factorization in polynomial feature space

Plenty of methods have been proposed in order to discover latent variables (features) in data sets. Such approaches include the principal component analysis (PCA), independent component analysis (ICA), factor analysis (FA), etc., to mention only a few. A recently investigated approach to decompose a data set with a given dimensionality into a lower dimensional space is the so-called nonnegative matrix factorization (NMF). Its only requirement is that both decomposition factors are nonnegative. To approximate the original data, the minimization of the NMF objective function is performed in the Euclidean space, where the difference between the original data and the factors can be minimized by employing L(2)-norm. In this paper, we propose a generalization of the NMF algorithm by translating the objective function into a Hilbert space (also called feature space) under nonnegativity constraints. With the help of kernel functions, we developed an approach that allows high-order dependencies between the basis images while keeping the nonnegativity constraints on both basis images and coefficients. Two practical applications, namely, facial expression and face recognition, show the potential of the proposed approach.     

基于Pearson系统模型的盲信号分离研究

独立分量分析是盲源分离的主流技术.自然梯度算法是其中非常重要的算法之一.介绍一种最大似然框架下的Pearson系统模型.该方法的优点是无须知晓信号的概率分布,实验结果表明,该算法能有效地分离随机混合的信号,特别对于非对称源有比同类算法更理想的效果.     

基于平滑l0范数正交子空间非负矩阵分解

为了能够提升分解矩阵的稀疏表达能力,提出了一种新的基于平滑l0范数的正交子空间非负矩阵分解方法。通过将分解矩阵的正交性及平滑l0范数约束同时引入矩阵分解的目标函数中一起进行优化,大大降低了计算复杂度,并提升了分解矩阵的稀疏表达能力。同时给出了分解矩阵的乘积更新迭代规则。通过在三个真实数据库(Iris,UCI,ORL)上的实验表明,该方法在分解所得矩阵的稀疏表示方面及将其应用于聚类问题所取得的聚类效果方面优于其他方法。     

Blind separation of uniformly distributed signals: a generalapproach

A general algorithm for blind separation of uniformly distributed signals is presented. First, maximum likelihood equations are obtained for dealing with this task. It is difficult to obtain a closed form maximum likelihood solution for arbitrary mixing matrix. The learning rules are obtained based on the geometric interpretation of the maximum likelihood estimator. The algorithm, under special constraint of orthogonal mixing matrix, is the same as the O(1/T(2)) convergent algorithm. Special noise correction mechanisms are incorporated in the algorithm, and it has been found that the algorithm exhibits stable performance even in the presence of large amount of noise.     

改进的快速独立分量分析方法研究

快速独立分量分析算法硬件实现困难,基于Huber M估计函数的独立分量算法硬件实现容易,但是该算法稳健性有待提高。针对这一问题,提出了一种硬件实现容易,而且具有很好稳健性的改进的快速独立分量分析算法。该算法硬件实现容易,且具有很好稳健性的Tukey双权函数作为原算法代价函数中的非线性函数,通过牛顿迭代方法对代价函数进行优化,得到改进的快速独立分量分析算法。仿真实验证明,该算法是有效的,且稳健性较好。     

Source number estimation and separation algorithms of underdetermined blind separation

Recently, sparse component analysis （SCA） has become a hot spot in BSS research. Instead of independent component analysis （ICA）, SCA can be used to solve underdetermined mixture efficiently. Two-step approach （TSA） is one of the typical methods to solve SCA based BSS problems. It estimates the mixing matrix before the separation of the sources. K-means clustering is often used to estimate the mixing matrix. It relies on the prior knowledge of the source number strongly. However, the estimation of the source number is an obstacle. In this paper, a fuzzy clustering method is proposed to estimate the source number and mixing matrix simultaneously. After that, the sources are recovered by the shortest path method （SPM）. Simulations show the availability and robustness of the proposed method.