On the relation between discriminant analysis and mutual information for supervised linear feature extraction

This paper provides a unifying view of three discriminant linear feature extraction methods: linear discriminant analysis, heteroscedastic discriminant analysis and maximization of mutual information. We propose a model-independent reformulation of the criteria related to these three methods that stresses their similarities and elucidates their differences. Based on assumptions for the probability distribution of the classification data, we obtain sufficient conditions under which two or more of the above criteria coincide. It is shown that these conditions also suffice for Bayes optimality of the criteria. Our approach results in an information-theoretic derivation of linear discriminant analysis and heteroscedastic discriminant analysis. Finally, regarding linear discriminant analysis, we discuss its relation to multidimensional independent component analysis and derive suboptimality bounds based on information theory.     

融合互信息与线性变换的非线性特征提取

由于线性变换无法较好保留数据的非线性结构而非线性变换往往需要进行大量的复杂运算,提出一种快速、高效的非线性特征提取方法。该方法通过研究互信息梯度在核空间中的线性不变性,采用互信息二次熵快速算法及梯度上升寻优策略,在有效降低计算量的同时能够提取有判别力的非线性高阶统计量。详细的数据投影和分类实验表明该方法在分类性能和算法时间复杂度上都优于传统算法。     

一种可最优化计算特征规模的互信息特征提取

利用矩阵特征向量分解,提出一种可最优化计算特征规模的互信息特征提取方法．首先,论述了高斯分布假设下的该互信息判据的类可分特性,并证明了现有典型算法都是本算法的特例;然后,在给出该互信息判据严格的数学意义基础上,提出了基于矩阵特征向量分解计算最优化特征规模算法;最后,通过实际数据验证了该方法的有效性     

一种互信息梯度不变的非线性特征提取方法*

针对线性的互信息特征提取方法,通过研究互信息梯度在核空间中的线性不变性,提出一种快速、高效的非线性特征提取方法。该方法采用互信息二次熵快速算法及梯度上升的寻优策略,提取有判别能力的非线性高阶统计量;在计算时避免传统非线性特征提取中的特征值分解运算,有效降低计算量。通过UCT数据的投影和分类实验表明,该方法无论在投影空间的可分性上,还是在算法时间复杂度上,都明显优于传统算法。     

Maximization of mutual information for offline Thai handwriting recognition

This paper aims to improve the performance of an HMM-based offline Thai handwriting recognition system through discriminative training and the use of fine-tuned feature extraction methods. The discriminative training is implemented by maximizing the mutual information between the data and their classes. The feature extraction is based on our proposed block-based PCA and composite images, shown to be better at discriminating Thai confusable characters. We demonstrate significant improvements in recognition accuracies compared to the classifiers that are not discriminatively optimized.     

Local structure based supervised feature extraction

In the past few years, the computer vision and pattern recognition community has witnessed the rapid growth of a new kind of feature extraction method, the manifold learning methods, which attempt to project the original data into a lower dimensional feature space by preserving the local neighborhood structure. Among them, locality preserving projection (LPP) is one of the most promising feature extraction techniques. However, when LPP is applied to the classification tasks, it shows some limitations, such as the ignorance of the label information. In this paper, we propose a novel feature extraction method, called locally discriminating projection (LDP). LDP utilizes class information to guide the procedure of feature extraction. In LDP, the local structure of the original data is constructed according to a certain kind of similarity between data points, which takes special consideration of both the local information and the class information. The similarity has several good properties which help to discover the true intrinsic structure of the data, and make LDP a robust technique for the classification tasks. We compare the proposed LDP approach with LPP, as well as other feature extraction methods, such as PCA and LDA, on the public available data sets, FERET and AR. Experimental results suggest that LDP provides a better representation of the class information and achieves much higher recognition accuracies.     

Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection

To recognize expressions accurately, facial expression systems require robust feature extraction and feature selection methods. In this paper, a normalized mutual information based feature selection technique is proposed for FER systems. The technique is derived from an existing method, that is, the max-relevance and min-redundancy (mRMR) method. We, however, propose to normalize the mutual information used in this method so that the domination of the relevance or of the redundancy can be eliminated. For feature extraction, curvelet transform is used. After the feature extraction and selection the feature space is reduced by employing linear discriminant analysis (LDA). Finally, hidden Markov model (HMM) is used to recognize the expressions. The proposed FER system (CNF-FER) is validated using four publicly available standard datasets. For each dataset, 10-fold cross validation scheme is utilized. CNF-FER outperformed the existing well-known statistical and state-of-the-art methods by achieving a weighted average recognition rate of 99 % across all the datasets.     

基于互信息的主成分分析特征选择算法

主成分分析是一种常用的特征选择算法,经典方法是计算各个特征之间的相关,但是相关无法评估变量间的非线性关系.互信息可用于衡量两个变量间相互依赖的强弱程度,且不局限于线性相关,鉴于此,提出一种基于互信息的主成分分析特征选择算法.该算法计算特征间的互信息,以互信息矩阵的特征值作为评价准则确定主成分的个数,并衡量主成分分析特征选择的效果.通过实例对所提出方法和传统主成分分析方法进行比较,并以神经网络为分类器分析分类效果.     

优化的互信息特征选择方法

在文本分类中,互信息是一种被广泛应用的特征选择方法,但是该方法仅考虑了特征的文档频而没有考虑特征的词频,导致它经常倾向于选择出现频率较低的特征。为此,提出了一个新的文档频并把它引入到互信息方法中,从而获得了一种优化的互信息方法。该优化的互信息方法不但考虑了特征的文档频而且还考虑了特征出现的词频。实验结果表明该优化的互信息方法性能良好。     

Image super-resolution using supervised multi-scale feature extraction network

Multimedia Tools and Applications - Image super-resolution using deep convolutional networks have recently achieved great successes. However, previous studies have failed to consider the spatial...     

Using clustering and dynamic mutual information for topic feature selection

A good feature selection method should take into account both category information and high‐frequency information to select useful features that can effectively display the information of a target. Because basic mutual information (BMI) prefers low‐frequency features and ignores high‐frequency features, clustering mutual information is proposed, which is based on clustering and makes effective high‐frequency features become unique, better integrating category information and useful high‐frequency information. Time is an important factor in topic detection and tracking (TDT). In order to improve the performance of TDT, time difference is integrated into clustering mutual information to dynamically adjust the mutual information, and then another algorithm called the dynamic clustering mutual information (DCMI) is given. In order to obtain the optimal subsets to display topics information, an objective function is proposed, which is based on the idea that a good feature subset should have the smallest distance within‐class and the largest distance across‐class. Experiments on TDT4 corpora using this objective function are performed; then, comparing the performances of BMI, DCMI, and the only existed topic feature selection algorithm Incremental Term Frequency‐Inverted Document Frequency (ITF‐IDF), these performance information will be displayed by four figures. Computation time of DCMI is previously lower than BMI and ITF‐IDF. The optimal normalized‐detection performance (C_det)_norm of DCMI is decreased by 0.3044 and 0.0970 compared with those of BMI and ITF‐IDF, respectively.     

基于互信息的文本特征加权方法

特征加权是文本分类中的重要环节,通过考察传统的特征选择函数,发现互信息方法在特征加权过程中表现尤为突出。为了提高互信息方法在特征加权时的性能,加入了词频信息、文档频率信息以及类别相关度因子,提出了一种基于改进的互信息特征加权方法。实验结果表明,该方法比传统的特征加权方法具有更好的分类性能。     

基于互信息的多级特征选择算法

针对在特征选择中选取特征较多时造成的去冗余过程很复杂的问题,以及一些特征需与其他特征组合后才会与标签有较强相关度的问题,提出了一种基于互信息的多级特征选择算法（MI_MLFS）。首先,根据特征与标签的相关度,将特征分为强相关、次强相关和其他特征;其次,选取强相关特征后,在次强相关特征中,选取冗余度较低的特征;最后,选取能增强已选特征集合与标签相关度的特征。在15组数据集上,将MI_MLFS与ReliefF、最大相关最小冗余（mRMR）算法、基于联合互信息（JMI）算法、条件互信息最大化准则（CMIM）算法和双输入对称关联（DISR）算法进行对比实验,结果表明MI_MLFS在支持向量机（SVM）和分类回归树（CART）分类器上分别有13组和11组数据集获得了最高的分类准确率。相较多种经典特征选择方法,MI_MLFS算法有更好的分类性能。     

基于互信息的多级特征选择算法

针对在特征选择中选取特征较多时造成的去冗余过程很复杂的问题，以及一些特征需与其他特征组合后才会与标签有较强相关度的问题，提出了一种基于互信息的多级特征选择算法（MI_MLFS）。首先，根据特征与标签的相关度，将特征分为强相关、次强相关和其他特征；其次，选取强相关特征后，在次强相关特征中，选取冗余度较低的特征；最后，选取能增强已选特征集合与标签相关度的特征。在15组数据集上，将MI_MLFS与ReliefF、最大相关最小冗余（mRMR）算法、基于联合互信息（JMI）算法、条件互信息最大化准则（CMIM）算法和双输入对称关联（DISR）算法进行对比实验，结果表明MI_MLFS在支持向量机（SVM）和分类回归树（CART）分类器上分别有13组和11组数据集获得了最高的分类准确率。相较多种经典特征选择方法，MI_MLFS算法有更好的分类性能。     

Adaptive linear discriminant analysis for online feature extraction

In this paper, we present new adaptive algorithms for the computation of the square root of the inverse covariance matrix. In contrast to the current similar methods, these new algorithms are obtained from an explicit cost function that is introduced for the first time. The new adaptive algorithms are used in a cascade form with a well-known adaptive principal component analysis to construct linear discriminant features. The adaptive nature and fast convergence rate of the new adaptive linear discriminant analysis algorithms make them appropriate for online pattern recognition applications. All adaptive algorithms discussed in this paper are trained simultaneously using a sequence of random data. Experimental results using the synthetic and real multiclass, multidimensional input data demonstrate the effectiveness of the new adaptive algorithms to extract the optimal features for the purpose of classification.     

Multiple-rank supervised canonical correlation analysis for feature extraction,fusion and recognition

The traditional CCA and 2D-CCA algorithms are unsupervised multiple feature extraction methods. Hence, introducing the supervised information of samples into these methods should be able to promote the classification performance. In this paper, a novel method is proposed to carry out the multiple feature extraction for classification, called two-dimensional supervised canonical correlation analysis (2D-SCCA), in which the supervised information is added to the criterion function. Then, by analyzing the relationship between GCCA and 2D-SCCA, another feature extraction method called multiple-rank supervised canonical correlation analysis (MSCCA) is also developed. Different from 2D-SCCA, in MSCCA k pairs left transforms and k pairs right transforms are sought to maximize the correlation. The convergence behavior and computational complexity of the algorithms are analyzed. Experimental results on real-world databases demonstrate the viability of the formulation, they also show that the classification results of our methods are higher than the other’s and the computing time is competitive. In this manner, the proposed methods proved to be the competitive multiple feature extraction and classification methods. As such, the two methods may well help to improve image recognition tasks, which are essential in many advanced expert and intelligent systems.     

Multi-label feature selection based on neighborhood mutual information

Multi-label learning deals with data associated with a set of labels simultaneously. Like traditional single-label learning, the high-dimensionality of data is a stumbling block for multi-label learning. In this paper, we first introduce the margin of instance to granulate all instances under different labels, and three different concepts of neighborhood are defined based on different cognitive viewpoints. Based on this, we generalize neighborhood information entropy to fit multi-label learning and propose three new measures of neighborhood mutual information. It is shown that these new measures are a natural extension from single-label learning to multi-label learning. Then, we present an optimization objective function to evaluate the quality of the candidate features, which can be solved by approximating the multi-label neighborhood mutual information. Finally, extensive experiments conducted on publicly available data sets verify the effectiveness of the proposed algorithm by comparing it with state-of-the-art methods.     

一种改进的基于条件互信息的特征选择算法

目前在文本分类领域较常用到的特征选择算法中,仅仅考虑了特征与类别之间的关联性,而对特征与特征之间的关联性没有予以足够的重视,这导致了特征之间预测能力的相互削弱,无法选出最有效的特征。提出了一种新的用于文本分类的特征选择算法（CMIM）,它可以帮助选出区分能力强、弱相关的特征。经实验验证,CMIM比传统的特征选择算法具有更好的性能。     

An improved feature transformation method using mutual information

The feature transformation is a very important step in pattern recognition systems. A feature transformation matrix can be obtained using different criteria such as discrimination between classes or feature independence or mutual information between features and classes. The obtained matrix can also be used for feature reduction. In this paper, we propose a new method for finding a feature transformation-based on Mutual Information (MI). For this purpose, we suppose that the Probability Density Function (PDF) of features in classes is Gaussian, and then we use the gradient ascent to maximize the mutual information between features and classes. Experimental results show that the proposed MI projection consistently outperforms other methods for a variety of cases. In the UCI Glass database we improve the classification accuracy up to 7.95 %. Besides, the improvement of phoneme recognition rate is 3.55 % on TIMIT.     

基于专家特征的条件互信息多标记特征选择算法

特征选择对于分类器的分类精度和泛化性能起重要作用。目前的多标记特征选择算法主要利用最大相关性最小冗余性准则在全部特征集中进行特征选择,没有考虑专家特征,因此多标记特征选择算法的运行时间较长、复杂度较高。实际上,在现实生活中专家依据几个或者多个关键特征就能够直接决定整体的预测方向。如果提取关注这些信息,必将减少特征选择的计算时间,甚至提升分类器性能。基于此,提出一种基于专家特征的条件互信息多标记特征选择算法。首先将专家特征与剩余的特征相联合,再利用条件互信息得出一个与标记集合相关性由强到弱的特征序列,最后通过划分子空间去除冗余性较大的特征。该算法在7个多标记数据集上进行了实验对比,结果表明该算法较其他特征选择算法有一定优势,统计假设检验与稳定性分析进一步证明了所提出算法的有效性和合理性。