Fast and robust fixed-point algorithms for independent componentanalysis

Independent component analysis (ICA) is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible. We use a combination of two different approaches for linear ICA: Comon's information theoretic approach and the projection pursuit approach. Using maximum entropy approximations of differential entropy, we introduce a family of new contrast functions for ICA. These contrast functions enable both the estimation of the whole decomposition by minimizing mutual information, and estimation of individual independent components as projection pursuit directions. The statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model, and it is shown how to choose contrast functions that are robust and/or of minimum variance. Finally, we introduce simple fixed-point algorithms for practical optimization of the contrast functions     

Advanced search algorithms for information-theoretic learning with kernel-based estimators

Recent publications have proposed various information-theoretic learning (ITL) criteria based on Renyi's quadratic entropy with nonparametric kernel-based density estimation as alternative performance metrics for both supervised and unsupervised adaptive system training. These metrics, based on entropy and mutual information, take into account higher order statistics unlike the mean-square error (MSE) criterion. The drawback of these information-based metrics is the increased computational complexity, which underscores the importance of efficient training algorithms. In this paper, we examine familiar advanced-parameter search algorithms and propose modifications to allow training of systems with these ITL criteria. The well known algorithms tailored here for ITL include various improved gradient-descent methods, conjugate gradient approaches, and the Levenberg-Marquardt (LM) algorithm. Sample problems and metrics are presented to illustrate the computational efficiency attained by employing the proposed algorithms.     

One-bit-matching conjecture for independent component analysis

The one-bit-matching conjecture for independent component analysis (ICA) could be understood from different perspectives but is basically stated as "all the sources can be separated as long as there is a one-to-one same-sign-correspondence between the kurtosis signs of all source probability density functions (pdf's) and the kurtosis signs of all model pdf's" (Xu, Cheung, & Amari, 1998a). This conjecture has been widely believed in the ICA community and implicitly supported by many ICA studies, such as the Extended Infomax (Lee, Girolami, & Sejnowski, 1999) and the soft switching algorithm (Welling & Weber, 2001). However, there is no mathematical proof to confirm the conjecture theoretically. In this article, only skewness and kurtosis are considered, and such a mathematical proof is given under the assumption that the skewness of the model densities vanishes. Moreover, empirical experiments are demonstrated on the robustness of the conjecture as the vanishing skewness assumption breaks. As a by-product, we also show that the kurtosis maximization criterion (Moreau & Macchi, 1996) is actually a special case of the minimum mutual information criterion for ICA.     

用于非高斯系统降维的最小残差熵主元网络

基于最小均方误差的主元分析和主元神经网络是有效的多变量降维统计技术,它们所提取的主元含有系统最大方差.非高斯随机系统的近似模型应当含有系统最大信息熵,但包含最大方差并不一定包含最大信息熵.该文提出一种以最小残差熵为通用指标的非线性主元神经网络模型,并给出了一种基于Parzen窗口密度函数估计的熵近似计算方法和网络学习算法.然后从信息论角度分析了,在高斯随机系统中基于最小残差熵和最小均方差为指标的主元网络学习结果具有一致性.最后以仿真验证该方法的有效性,并与基于最小均方误差的主元分析和主元神经网络方法的计算结果进行对比性分析.     

Learning the higher-order structure of a natural sound

Unsupervised learning algorithms paying attention only to second-order statistics ignore the phase structure (higher-order statistics) of signals, which contains all the informative temporal and spatial coincidences which we think of as 'features'. Here we discuss how an Independent Component Analysis (ICA) algorithm may be used to elucidate the higher-order structure of natural signals, yielding their independent basis functions. This is illustrated with the ICA transform of the sound of a fingernail tapping musically on a tooth. The resulting independent basis functions look like the sounds themselves, having similar temporal envelopes and the same musical pitches. Thus they reflect both the phase and frequency information inherent in the data.     

Measuring relevance between discrete and continuous features based on neighborhood mutual information

Measures of relevance between features play an important role in classification and regression analysis. Mutual information has been proved an effective measure for decision tree construction and feature selection. However, there is a limitation in computing relevance between numerical features with mutual information due to problems of estimating probability density functions in high-dimensional spaces. In this work, we generalize Shannon’s information entropy to neighborhood information entropy and propose a measure of neighborhood mutual information. It is shown that the new measure is a natural extension of classical mutual information which reduces to the classical one if features are discrete; thus the new measure can also be used to compute the relevance between discrete variables. In addition, the new measure introduces a parameter delta to control the granularity in analyzing data. With numeric experiments, we show that neighborhood mutual information produces the nearly same outputs as mutual information. However, unlike mutual information, no discretization is required in computing relevance when used the proposed algorithm. We combine the proposed measure with four classes of evaluating strategies used for feature selection. Finally, the proposed algorithms are tested on several benchmark data sets. The results show that neighborhood mutual information based algorithms yield better performance than some classical ones.     

基于最大类间后验交叉熵的阈值化分割算法

从目标和背景的类间差异性出发,提出了一种基于最大类间交叉熵准则的阈值化分割新算法。该算法假设目标和背景象素的条件分布服从正态分布,利用贝叶斯公式估计象素属于目标和背景两类区域的后验概率,再搜索这两类区域后验概率之间的最大交叉熵。比较了新算法与基于最小交叉熵以及基于传统香农熵的阈值化算法的特点和分割性能。     

随机自适应控制的信息论方法

从Ｓｈａｎｎｏｎ信息理论的角度，分别应用最小熵方法和最大互信息方法，对摸型参数不确定的随机系统的自适应控制问题进行了研究和比较．对于这类系统，由最大互信息方法导出的自适应控制律本质上具有双重控制的特性．     

基于最大类间后验交叉熵的阈值比分割算法

从目标和背景的类间差异性出发,提出了一种基于最大类间交叉熵准则的阈值化分割新算法,算法阈设目标的背景象素的条件分布服从正态分布,利用贝叶期公式估计象素属于目标和背景两类区域的后验概率,再搜索这两为区域后验概率之间的最大交叉熵。比较了新算法一基于最小交叉熵以及基于传统香农熵的阈值化算法的分割性能。     

ICA mixture model algorithm for unsupervised classification of remote sensing imagery

Conventional remote sensing classification algorithms assume that the data in each class can be modelled using a multivariate Gaussian distribution. As this assumption is often not valid in practice, conventional algorithms do not perform well. In this paper, we present an independent component analysis (ICA)‐based approach for unsupervised classification of multi/hyperspectral imagery. ICA used for a mixture model estimates the data density in each class and models class distributions with non‐Gaussian (sub‐ and super‐Gaussian) probability density functions, resulting in the ICA mixture model (ICAMM) algorithm. Independent components and the mixing matrix for each class are found using an extended information‐maximization algorithm, and the class membership probabilities for each pixel are computed. The pixel is allocated to the class having maximum class membership probability to produce a classification. We apply the ICAMM algorithm for unsupervised classification of images obtained from both multispectral and hyperspectral sensors. Four feature extraction techniques are considered as a preprocessing step to reduce the dimensionality of the hyperspectral data. The results demonstrate that the ICAMM algorithm significantly outperforms the conventional K‐means algorithm for land cover classification produced from both multi‐ and hyperspectral remote sensing images.     

基于最大互信息最大相关熵的特征选择方法

特征选择算法主要分为filter和wrapper两大类,并已提出基于不同理论的算法模型,但依然存在算法处理能力不强、子集分类精度不高等问题。基于模糊粗糙集的信息熵模型提出最大互信息最大相关熵标准,并根据该标准设计了一种新的特征选择方法,能同时处理离散数据、连续数据和模糊数据等混合信息。经UCI数据集试验,表明该算法与其他算法相比,具有较高的精度,且稳定性较高,是有效的。     

A new independent component analysis for speech recognition and separation

This paper presents a novel nonparametric likelihood ratio (NLR) objective function for independent component analysis (ICA). This function is derived through the statistical hypothesis test of independence of random observations. A likelihood ratio function is developed to measure the confidence toward independence. We accordingly estimate the demixing matrix by maximizing the likelihood ratio function and apply it to transform data into independent component space. Conventionally, the test of independence was established assuming data distributions being Gaussian, which is improper to realize ICA. To avoid assuming Gaussianity in hypothesis testing, we propose a nonparametric approach where the distributions of random variables are calculated using kernel density functions. A new ICA is then fulfilled through the NLR objective function. Interestingly, we apply the proposed NLR-ICA algorithm for unsupervised learning of unknown pronunciation variations. The clusters of speech hidden Markov models are estimated to characterize multiple pronunciations of subword units for robust speech recognition. Also, the NLR-ICA is applied to separate the linear mixture of speech and audio signals. In the experiments, NLR-ICA achieves better speech recognition performance compared to parametric and nonparametric minimum mutual information ICA.     

Complex ICA by negentropy maximization.

In this paper, we use complex analytic functions to achieve independent component analysis (ICA) by maximization of non-Gaussianity and introduce the complex maximization of non-Gaussianity (CMN) algorithm. We derive both a gradient-descent and a quasi-Newton algorithm that use the full second-order statistics providing superior performance with circular and noncircular sources as compared to existing methods. We show the connection among ICA methods through maximization of non-Gaussianity, mutual information, and maximum likelihood (ML) for the complex case, and emphasize the importance of density matching for all three cases. Local stability conditions are derived for the CMN cost function that explicitly show the effects of noncircularity on convergence and demonstrated through simulation examples.     

维吾尔文无监督自动切分及无监督特征选择

维吾尔文常用切分方法会产生大量的语义抽象甚至多义的词特征,因此学习算法难以发现高维数据中隐藏的结构.提出一种无监督切分方法dme-TS和一种无监督特征选择方法UMRMR-UFS.dme-TS从大规模生语料中自动获取单词Bi-gram及上下文语境信息,并将相邻单词间的t-测试差、互信息及双词上下文邻接对熵的线性融合作为一个组合统计量(dme)来评价单词间的结合能力,从而将文本切分成语义具体的独立语言单位的特征集合.UMRMR-UFS用一种综合考虑最大相关度和最小冗余的无监督特征选择标准(UMRMR)来评价每一个特征的重要性,并将最重要的特征依次移入到特征子集中.实验结果表明dme-TS能有效控制原始特征集的规模,提高特征项本身的质量,用UMRMR-UFS的输出来表征文本时,学习算法也表现出其最高的性能.     

Audio–Visual Affective Expression Recognition Through Multistream Fused HMM

Advances in computer processing power and emerging algorithms are allowing new ways of envisioning human-computer interaction. Although the benefit of audio-visual fusion is expected for affect recognition from the psychological and engineering perspectives, most of existing approaches to automatic human affect analysis are unimodal: information processed by computer system is limited to either face images or the speech signals. This paper focuses on the development of a computing algorithm that uses both audio and visual sensors to detect and track a user's affective state to aid computer decision making. Using our multistream fused hidden Markov model (MFHMM), we analyzed coupled audio and visual streams to detect four cognitive states (interest, boredom, frustration and puzzlement) and seven prototypical emotions (neural, happiness, sadness, anger, disgust, fear and surprise). The MFHMM allows the building of an optimal connection among multiple streams according to the maximum entropy principle and the maximum mutual information criterion. Person-independent experimental results from 20 subjects in 660 sequences show that the MFHMM approach outperforms face-only HMM, pitch-only HMM, energy-only HMM, and independent HMM fusion, under clean and varying audio channel noise condition.     

基于CEEMDAN-ICA的单通道脑电信号眼电伪迹滤除方法

传统盲源分离法不能解决欠定问题,且分离信号与源信号对应关系不确定.提出一种基于自适应噪声完备经验模态分解(CEEMDAN)和独立成分分析(ICA)相结合的脑电信号眼电伪迹自动去除方法.该方法首先将含伪迹脑电信号自适应分解成多维本征模态函数(IMF),以满足盲源分离方法对信号正定或超定要求,再对本征模态函数用ICA方法构建多维源信号,最后利用模糊熵阈值判据判别多维源信号中的伪迹信号,完成滤波并重构脑电信号.该方法相比于其他算法,能更好的去除眼电伪迹并保留原始信息,适合单通道脑电信号预处理.     

基于信息论的高维海量数据离群点挖掘

针对高维海量数据集离群点挖掘存在“维数灾难”的问题,提出了基于信息论的高维海量数据的离群点挖掘算法。该算法采用属性选择,去除冗余属性降维。利用信息嫡作为离群点判断的度量标准,消除距离和密度量纲的弊端。在真实数据集上的实验结果表明,算法对高维海量数据离群点挖掘是有效可行的,其效率和精度得到了明显提高。     

基于最小联合互信息亏损的最优特征选择算法

提出了一种基于最小联合互信息亏损的最优特征选择算法。该算法首先通过一种动态渐增策略搜索一个特征全集的无差异特征子集,并基于最小条件互信息原则在保证每一步中联合互信息量亏损都最小的情况下筛选其中的冗余特征,从而得到一个近似最优特征子集。针对现有基于条件互信息的条件独立性测试方法在高维特征域上所面临的效率瓶颈问题,给出了一种用于估计条件互信息的快速实现方法,并将其用于所提算法的实现。分类实验结果表明,所提算法优于经典的特征选择算法。此外,执行效率实验结果表明,所提条件互信息的快速实现方法在执行效率上有着显著的优势。     

区间序信息系统的无监督特征选择*

目前已有很多针对单值信息系统的无监督特征选择方法,但针对区间值信息系统的无监督特征选择方法却很少.针对区间序信息系统,文中提出模糊优势关系,并基于此关系扩展模糊排序信息熵和模糊排序互信息,用于评价特征的重要性.再结合一种综合考虑信息量和冗余度的无监督最大信息最小冗余(UmIMR)准则,构造无监督特征选择方法.最后通过实验证明文中方法的有效性.     

Objective priors from maximum entropy in data classification

Lack of knowledge of the prior distribution in classification problems that operate on small data sets may make the application of Bayes’ rule questionable. Uniform or arbitrary priors may provide classification answers that, even in simple examples, may end up contradicting our common sense about the problem. Entropic priors (EPs), via application of the maximum entropy (ME) principle, seem to provide good objective answers in practical cases leading to more conservative Bayesian inferences. EP are derived and applied to classification tasks when only the likelihood functions are available. In this paper, when inference is based only on one sample, we review the use of the EP also in comparison to priors that are obtained from maximization of the mutual information between observations and classes. This last criterion coincides with the maximization of the KL divergence between posteriors and priors that for large sample sets leads to the well-known reference (or Bernardo’s) priors. Our comparison on single samples considers both approaches in prospective and clarifies differences and potentials. A combinatorial justification for EP, inspired by Wallis’ combinatorial argument for entropy definition, is also included.The application of the EP to sequences (multiple samples) that may be affected by excessive domination of the class with the maximum entropy is also considered with a solution that guarantees posterior consistency. An explicit iterative algorithm is proposed for EP determination solely from knowledge of the likelihood functions. Simulations that compare EP with uniform priors on short sequences are also included.