A common neural-network model for unsupervised exploratory data analysis and independent component analysis.

This paper presents the derivation of an unsupervised learning algorithm, which enables the identification and visualization of latent structure within ensembles of high-dimensional data. This provides a linear projection of the data onto a lower dimensional subspace to identify the characteristic structure of the observations independent latent causes. The algorithm is shown to be a very promising tool for unsupervised exploratory data analysis and data visualization. Experimental results confirm the attractiveness of this technique for exploratory data analysis and an empirical comparison is made with the recently proposed generative topographic mapping (GTM) and standard principal component analysis (PCA). Based on standard probability density models a generic nonlinearity is developed which allows both (1) identification and visualization of dichotomised clusters inherent in the observed data and (2) separation of sources with arbitrary distributions from mixtures, whose dimensionality may be greater than that of number of sources. The resulting algorithm is therefore also a generalized neural approach to independent component analysis (ICA) and it is considered to be a promising method for analysis of real-world data that will consist of sub- and super-Gaussian components such as biomedical signals.     

A robust approach to independent component analysis of signals with high-level noise measurements

We propose a robust approach for independent component analysis (ICA) of signals where observations are contaminated with high-level additive noise and/or outliers. The source signals may contain mixtures of both sub-Gaussian and super-Gaussian components, and the number of sources is unknown. Our robust approach includes two procedures. In the first procedure, a robust prewhitening technique is used to reduce the power of additive noise, the dimensionality and the correlation among sources. A cross-validation technique is introduced to estimate the number of sources in this first procedure. In the second procedure, a nonlinear function is derived using the parameterized t-distribution density model. This nonlinear function is robust against the undue influence of outliers fundamentally. Moreover, the stability of the proposed algorithm and the robust property of misestimating the parameters (kurtosis) have been studied. By combining the t-distribution model with a family of light-tailed distributions (sub-Gaussian) model, we can separate the mixture of sub-Gaussian and super-Gaussian source components. Through the analysis of artificially synthesized data and real-world magnetoencephalographic (MEG) data, we illustrate the efficacy of this robust approach.     

Markov and Semi-Markov switching of source appearances for nonstationary independent component analysis.

Independent component analysis (ICA) is currently the most popularly used approach to blind source separation (BSS), the problem of recovering unknown source signals when their mixtures are observed but the actual mixing process is unknown. Many ICA algorithms assume that a fixed set of source signals consistently exists in mixtures throughout the time-series to be examined. However, real-world signals often have such difficult nonstationarity that each source signal abruptly appears or disappears, thus the set of active sources dynamically changes with time. In this paper, we propose switching ICA (SwICA), which focuses on such situations. The proposed approach is based on the noisy ICA formulated as a generative model. We employ a special type of hidden Markov model (HMM) to represent such prior knowledge that the source may abruptly appear or disappear with time. The special HMM setting t hen provides an effect ofvariable selection in a dynamic way. We use the variational Bayes (VB) method to derive an effective approximation of Bayesian inference for this model. In simulation experiments using artificial and realistic source signals, the proposed method exhibited performance superior to existing methods, especially in the presence of noise. The compared methods include the natural-gradient ICA with a nonholonomic constraint, and the existing ICA method incorporating an HMM source model, which aims to deal with general nonstationarities that may exist in source signals. In addition, the proposed method could successfully recover the source signals even when the total number of true sources was overestimated or was larger than that of mixtures. We also propose a modification of the basic Markov model into a semi-Markov model, and show that the semi-Markov one is more effective for robust estimation of the source appearance.     

Separation theorem for independent subspace analysis and its consequences

Independent component analysis (ICA) – the theory of mixed, independent, non-Gaussian sources – has a central role in signal processing, computer vision and pattern recognition. One of the most fundamental conjectures of this research field is that independent subspace analysis (ISA) – the extension of the ICA problem, where groups of sources are independent – can be solved by traditional ICA followed by grouping the ICA components. The conjecture, called ISA separation principle, (i) has been rigorously proven for some distribution types recently, (ii) forms the basis of the state-of-the-art ISA solvers, (iii) enables one to estimate the unknown number and the dimensions of the sources efficiently, and (iv) can be extended to generalizations of the ISA task, such as different linear-, controlled-, post nonlinear-, complex valued-, partially observed problems, as well as to problems dealing with nonparametric source dynamics. Here, we shall review the advances on this field.     

Dual multivariate auto-regressive modeling in state space for temporal signal separation

Many existing independent component analysis (ICA) approaches result in deteriorated performance in temporal source separation because they have not taken into consideration of the underlying temporal structure of sources. In this paper, we model temporal sources as a general multivariate auto-regressive (AR) process whereby an underlying multivariate AR process in observation space is obtained. In this dual AR modeling, the mixing process from temporal sources to observations is the same as the mixture from the nontemporal residuals of the source AR (SAR) process to that of the observation AR (OAR) process. We can therefore avoid the source temporal effects in performing ICA by learning the demixing system on the independently distributed OAR residuals rather than the time-correlated observations. Particularly, we implement this approach by modeling each source signal as a finite mixture of generalized autoregressive conditional heteroskedastic (GARCH) process. The adaptive algorithms are proposed to extract the OAR residuals appropriately online, together with learning the demixing system via a nontemporal ICA algorithm. The experiments have shown its superior performance on temporal source separation.     

A new independent component analysis for speech recognition and separation

This paper presents a novel nonparametric likelihood ratio (NLR) objective function for independent component analysis (ICA). This function is derived through the statistical hypothesis test of independence of random observations. A likelihood ratio function is developed to measure the confidence toward independence. We accordingly estimate the demixing matrix by maximizing the likelihood ratio function and apply it to transform data into independent component space. Conventionally, the test of independence was established assuming data distributions being Gaussian, which is improper to realize ICA. To avoid assuming Gaussianity in hypothesis testing, we propose a nonparametric approach where the distributions of random variables are calculated using kernel density functions. A new ICA is then fulfilled through the NLR objective function. Interestingly, we apply the proposed NLR-ICA algorithm for unsupervised learning of unknown pronunciation variations. The clusters of speech hidden Markov models are estimated to characterize multiple pronunciations of subword units for robust speech recognition. Also, the NLR-ICA is applied to separate the linear mixture of speech and audio signals. In the experiments, NLR-ICA achieves better speech recognition performance compared to parametric and nonparametric minimum mutual information ICA.     

一种改进的FastICA算法及其应用

独立分量分析是基于信号高阶统计量的信号分析方法,它可以找到隐含在数据中的独立分量,已经广泛应用到语音信号处理、图像处理及信息通信等方面。目前应用较多的快速独立分量分析（FastICA）利用了牛顿迭代法原理,具有较快的收敛速度,但对初始值的选择比较敏感。为克服其缺点,改进其优化学习算法,在牛顿迭代方向增加一维搜索,使改进后的算法的收敛性不依赖于初始值的选择。将改进的FastICA算法应用到运动目标检测中,取得稳定性较强的结果。     

Feature extraction through LOCOCODE

Low-complexity coding and decoding (LOCOCODE) is a novel approach to sensory coding and unsupervised learning. Unlike previous methods, it explicitly takes into account the information-theoretic complexity of the code generator. It computes lococodes that convey information about the input data and can be computed and decoded by low-complexity mappings. We implement LOCOCODE by training autoassociators with flat minimum search, a recent, general method for discovering low-complexity neural nets. It turns out that this approach can unmix an unknown number of independent data sources by extracting a minimal number of low-complexity features necessary for representing the data. Experiments show that unlike codes obtained with standard autoencoders, lococodes are based on feature detectors, never unstructured, usually sparse, and sometimes factorial or local (depending on statistical properties of the data). Although LOCOCODE is not explicitly designed to enforce sparse or factorial codes, it extracts optimal codes for difficult versions of the "bars" benchmark problem, whereas independent component analysis (ICA) and principal component analysis (PCA) do not. It produces familiar, biologically plausible feature detectors when applied to real-world images and codes with fewer bits per pixel than ICA and PCA. Unlike ICA, it does not need to know the number of independent sources. As a preprocessor for a vowel recognition benchmark problem, it sets the stage for excellent classification performance. Our results reveal an interesting, previously ignored connection between two important fields: regularizer research and ICA-related research. They may represent a first step toward unification of regularization and unsupervised learning.     

Decompositional independent component analysis using multi-objective optimization

Current approaches for blind source separation, such as independent component analysis (ICA), implicitly assume that the number of collected signals equals the number of sources. This assumption does not hold true in many real-world applications as in the case of electroencephalographic (EEG) data collected from the surface of a human’s scalp, where independent EEG information is mixed with independent artifacts. This situation is abstracted in this paper by introducing the singers’ party problem, where the number of signals collected from the party equals the number of singers. However, there are also a number of instruments playing at the party representing independent sources that need to be removed correctly to extract the voices of the singers. In this paper, we introduce a decompositional approach to project the sources found in ICA into a higher-dimensional space; providing the ability to separate local (singers) information from shared/global (instruments) information. The decomposition will also associate each component with a mixed signal, creating a bijective relationship between the mixed signals and the sources. The problem is formulated as a multi-objective optimization problem. We compare the pros and cons of two different multi-objective formulations of the problem and demonstrate that one of the formulations can effectively solve the singers party problem.     

Information filtering using SVD and ICA

We propose an information filtering system for documents by a user profile using latent semantics obtained by singular value decomposition (SVD) and independent component analysis (ICA). In information filtering systems, it is useful to analyze the latent semantics of documents. ICA is one method to analyze the latent semantics. We assume that topics are independent of each other. Hence, when ICA is applied to documents, we obtain the topics included in the documents. By using SVD remove noises before applying ICA, we can improve the accuracy of topic extraction. By representation of the documents with those topics, we improve recommendations by the user profile. In addition, we construct a user profile with a genetic algorithm (GA) and evaluate it by 11-point average precision. We carried out an experiment using a test collection to confirm the advantages of the proposed method. This work was presented in part at the 10th International Symposium on Artificial Life and Robotics, Oita, Japan, February 4–6, 2005     

Nonlinear and noisy extension of independent component analysis: theory and its application to a pitch sensation model

In this letter, we propose a noisy nonlinear version of independent component analysis (ICA). Assuming that the probability density function (p. d. f.) of sources is known, a learning rule is derived based on maximum likelihood estimation (MLE). Our model involves some algorithms of noisy linear ICA (e. g., Bermond & Cardoso, 1999) or noise-free nonlinear ICA (e. g., Lee, Koehler, & Orglmeister, 1997) as special cases. Especially when the nonlinear function is linear, the learning rule derived as a generalized expectation-maximization algorithm has a similar form to the noisy ICA algorithm previously presented by Douglas, Cichocki, and Amari (1998). Moreover, our learning rule becomes identical to the standard noise-free linear ICA algorithm in the noiseless limit, while existing MLE-based noisy ICA algorithms do not rigorously include the noise-free ICA. We trained our noisy nonlinear ICA by using acoustic signals such as speech and music. The model after learning successfully simulates virtual pitch phenomena, and the existence region of virtual pitch is qualitatively similar to that observed in a psychoacoustic experiment. Although a linear transformation hypothesized in the central auditory system can account for the pitch sensation, our model suggests that the linear transformation can be acquired through learning from actual acoustic signals. Since our model includes a cepstrum analysis in a special case, it is expected to provide a useful feature extraction method that has often been given by the cepstrum analysis.     

基于协方差ICA分析的多重振荡源分离方法

提出一种新的基于协方差独立源分析（ICA）的多重振荡源分离定位方法.把控制系统中受到振荡干扰的过程数据变换到协方差函数,利用ICA分析的方法进行多重振荡源分离.通过仿真实验对比分析,指出其他时域主元分析（PCA） 、时域ICA、协方差PCA等方法的不足,而协方差ICA分析能够准确地分离并定位多重振荡干扰源.仿真结果表明该方法是可行的.     

Modular organization of adaptive colouration in flounder and cuttlefish revealed by independent component analysis

Flounders and cuttlefish have an impressive ability to change colouration, for camouflage and, in the case of cuttlefish, for communication. We pursue the hypothesis that these diverse patterns are created by combining a small number of distinct pattern modules. Independent component analysis (ICA) is a powerful tool for identifying independent sources of variation in linear mixtures of signals. Two versions of ICA are used, one assuming that sources have independence over time, and the other over space. These reveal the modularity of the skin colouration system, and suggest how the pattern modules are combined in specific behavioural contexts. ICA may therefore be a useful tool for studying animal camouflage and communication.     

Stochastic trapping in a solvable model of on-line independent component analysis.

Previous analytical studies of on-line independent component analysis (ICA) learning rules have focused on asymptotic stability and efficiency. In practice, the transient stages of learning are often more significant in determining the success of an algorithm. This is demonstrated here with an analysis of a Hebbian ICA algorithm, which can find a small number of nongaussian components given data composed of a linear mixture of independent source signals. An idealized data model is considered in which the sources comprise a number of nongaussian and gaussian sources, and a solution to the dynamics is obtained in the limit where the number of gaussian sources is infinite. Previous stability results are confirmed by expanding around optimal fixed points, where a closed-form solution to the learning dynamics is obtained. However, stochastic effects are shown to stabilize otherwise unstable suboptimal fixed points. Conditions required to destabilize one such fixed point are obtained for the case of a single nongaussian component, indicating that the initial learning rate eta required to escape successfully is very low (eta = O(N(-2)) where N is the data dimension), resulting in very slow learning typically requiring O(N(3)) iterations. Simulations confirm that this picture holds for a finite system.     

Lattice independent component analysis for functional magnetic resonance imaging

We introduce a lattice independent component analysis (LICA) unsupervised scheme to functional magnetic resonance imaging (fMRI) data analysis. LICA is a non-linear alternative to independent component analysis (ICA), such that ICA’s statistical independent sources correspond to LICA’s lattice independent sources. In this paper, LICA uses an incremental lattice source induction algorithm (ILSIA) to induce the lattice independent sources from the input dataset. The ILSIA computes a set of Strongly Lattice Independent vectors using properties of lattice associative memories regarding Lattice Independence and Chebyshev best approximation. The lattice independent sources constitute a set of Affine Independent vectors that define a simplex covering the input data. LICA carries out data linear unmixing based on the lattice independent sources basis. Therefore, LICA is a hybrid combination of a non-linear lattice based component and a linear unmixing component. The principal advantage over ICA is that LICA does not impose any probabilistic model assumptions on the data sources. We compare LICA with ICA in two case studies. Firstly, including simulated fMRI data, LICA discovers the spatial location of meaningful sources with less ambiguity than ICA. Secondly, including real data from an auditory stimulation experiment, LICA improves over some state of the art ICA variants discovering the activation patterns detected by Statistical Parametric Mapping (SPM) on the same data.     

Independent factor analysis

We introduce the independent factor analysis (IFA) method for recovering independent hidden sources from their observed mixtures. IFA generalizes and unifies ordinary factor analysis (FA), principal component analysis (PCA), and independent component analysis (ICA), and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy. IFA is a two-step procedure. In the first step, the source densities, mixing matrix, and noise covariance are estimated from the observed data by maximum likelihood. For this purpose we present an expectation-maximization (EM) algorithm, which performs unsupervised learning of an associated probabilistic model of the mixing situation. Each source in our model is described by a mixture of gaussians; thus, all the probabilistic calculations can be performed analytically. In the second step, the sources are reconstructed from the observed data by an optimal nonlinear estimator. A variational approximation of this algorithm is derived for cases with a large number of sources, where the exact algorithm becomes intractable. Our IFA algorithm reduces to the one for ordinary FA when the sources become gaussian, and to an EM algorithm for PCA in the zero-noise limit. We derive an additional EM algorithm specifically for noiseless IFA. This algorithm is shown to be superior to ICA since it can learn arbitrary source densities from the data. Beyond blind separation, IFA can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding.     

A hierarchical Bayesian model for learning nonlinear statistical regularities in nonstationary natural signals

Capturing statistical regularities in complex, high-dimensional data is an important problem in machine learning and signal processing. Models such as principal component analysis (PCA) and independent component analysis (ICA) make few assumptions about the structure in the data and have good scaling properties, but they are limited to representing linear statistical regularities and assume that the distribution of the data is stationary. For many natural, complex signals, the latent variables often exhibit residual dependencies as well as nonstationary statistics. Here we present a hierarchical Bayesian model that is able to capture higher-order nonlinear structure and represent nonstationary data distributions. The model is a generalization of ICA in which the basis function coefficients are no longer assumed to be independent; instead, the dependencies in their magnitudes are captured by a set of density components. Each density component describes a common pattern of deviation from the marginal density of the pattern ensemble; in different combinations, they can describe nonstationary distributions. Adapting the model to image or audio data yields a nonlinear, distributed code for higher-order statistical regularities that reflect more abstract, invariant properties of the signal.     

Phase Ambiguity Correction and Visualization Techniques for Complex-Valued ICA of Group fMRI Data

Analysis of functional magnetic resonance imaging (fMRI) data in its native, complex form has been shown to increase the sensitivity both for data-driven techniques, such as independent component analysis (ICA), and for model-driven techniques. The promise of an increase in sensitivity and specificity in clinical studies, provides a powerful motivation for utilizing both the phase and magnitude data; however, the unknown and noisy nature of the phase poses a challenge. In addition, many complex-valued analysis algorithms, such as ICA, suffer from an inherent phase ambiguity, which introduces additional difficulty for group analysis. We present solutions for these issues, which have been among the main reasons phase information has been traditionally discarded, and show their effectiveness when used as part of a complex-valued group ICA algorithm application. The methods we present thus allow the development of new fully complex data-driven and semi-blind methods to process, analyze, and visualize fMRI data.We first introduce a phase ambiguity correction scheme that can be either applied subsequent to ICA of fMRI data or can be incorporated into the ICA algorithm in the form of prior information to eliminate the need for further processing for phase correction. We also present a Mahalanobis distance-based thresholding method, which incorporates both magnitude and phase information into a single threshold, that can be used to increase the sensitivity in the identification of voxels of interest. This method shows particular promise for identifying voxels with significant susceptibility changes but that are located in low magnitude (i.e. activation) areas. We demonstrate the performance gain of the introduced methods on actual fMRI data.     

Can we learn anything from single-channel unaveraged MEG data?

A method for the decomposition of single-channel unaveraged magnetoencephalographic (MEG) data into statistically independent components is presented. The study of MEG recordings is characterised by a host of difficulties, most of which stem from the inherently noisy recording process by which the data is obtained. MEG time series typically contain a mix of artifactual components from a variety of sources, and the isolation of interesting signals from this noise background poses a difficult problem. In this article, we present a novel approach combining the techniques of independent component analysis (ICA) and dynamical embedding, which can be used to extract and isolate components of interest from single-channel unaveraged MEG data. In our approach, the method of delays is proposed as a means of augmenting the single-channel data, thus, facilitating the application of ICA. Finally, because the single-channel approach yields no information regarding the physiological origins of extracted sources, we discuss a method by which extracted sources may be projected back into the multichannel measurement space, permitting an estimate of the respective spatial distributions to be obtained. The proposed methods are tested on three separate MEG channels and the results are presented and discussed.     

基于ICA的周期性噪声消除算法

为了使问题有解，传统的独立分量分析算法对问题的条件有许多严格的限制，其中包括观测信号的个数不能小于源信号的个数等．在降噪等实际应用中，观测信号的个数可能无法满足这一条件，为了能够利用独立分量分析分离加性噪声，需要人工构造混合信号．基于周期性干扰表现的整体周期性，提出了一种构造混合信号的新算法．利用构造的混合信号进行独立分量分析，可以有效地消除周期性干扰，使目标信号的信噪比显著提高．即使在信噪比很低，目标信号几近被“淹没”的情况下，仍然能够较好地将其分离出来．该方法具有算法简单、运算速度快、算法效率高等特点．计算机仿真和实验结果都证明了算法的有效性．