Fast kernel Fisher discriminant analysis via approximating the kernel principal component analysis

Kernel Fisher discriminant analysis (KFDA) extracts a nonlinear feature from a sample by calculating as many kernel functions as the training samples. Thus, its computational efficiency is inversely proportional to the size of the training sample set. In this paper we propose a more approach to efficient nonlinear feature extraction, FKFDA (fast KFDA). This FKFDA consists of two parts. First, we select a portion of training samples based on two criteria produced by approximating the kernel principal component analysis (AKPCA) in the kernel feature space. Then, referring to the selected training samples as nodes, we formulate FKFDA to improve the efficiency of nonlinear feature extraction. In FKFDA, the discriminant vectors are expressed as linear combinations of nodes in the kernel feature space, and the extraction of a feature from a sample only requires calculating as many kernel functions as the nodes. Therefore, the proposed FKFDA has a much faster feature extraction procedure compared with the naive kernel-based methods. Experimental results on face recognition and benchmark datasets classification suggest that the proposed FKFDA can generate well classified features.     

Two realizations of a general feature extraction framework

A general feature extraction framework is proposed as an extension of conventional linear discriminant analysis. Two nonlinear feature extraction algorithms based on this framework are investigated. The first is a kernel function feature extraction (KFFE) algorithm. A disturbance term is introduced to regularize the algorithm. Moreover, it is revealed that some existing nonlinear feature extraction algorithms are the special cases of this KFFE algorithm. The second feature extraction algorithm, mean-STD¹-norm feature extraction algorithm, is also derived from the framework. Experiments based on both synthetic and real data are presented to demonstrate the performance of both feature extraction algorithms.     

Kernel-view based discriminant approach for embedded feature extraction in high-dimensional space

Derived from the traditional manifold learning algorithms, local discriminant analysis methods identify the underlying submanifold structures while employing discriminative information for dimensionality reduction. Mathematically, they can all be unified into a graph embedding framework with different construction criteria. However, such learning algorithms are limited by the curse-of-dimensionality if the original data lie on the high-dimensional manifold. Different from the existing algorithms, we consider the discriminant embedding as a kernel analysis approach in the sample space, and a kernel-view based discriminant method is proposed for the embedded feature extraction, where both PCA pre-processing and the pruning of data can be avoided. Extensive experiments on the high-dimensional data sets show the robustness and outstanding performance of our proposed method.     

A new feature extraction method for odour classification

A new method for real time classification of volatile chemical substance traces is presented. The method is based on electrochemical signals of an array of semiconductor gas sensors. In these sensor signals characteristic patterns of different substances are hidden. There are non-linear correlative relationships between the measured sensor signals and the chemical substances which are treated using two methods derived from statistical learning theory (Support Vector Machine - SVM, Maximum Likelihood Estimation - MLE) for the detection of the substance characteristics in the sensor signals. A key criterion for the presented pattern recognition is a newly developed type of features, which is specially adapted to the low frequency signals of semiconductor sensors. The presented features are based on the evaluation of the range of the transient response in the sensor signals in the frequency domain.To derive the new features, both real measurement data and synthetic generated signals were used. In the experiments the focus was set on the creation of reproducible sensor signals to get characteristic signal patterns. Synthetic signals were derived from a Gaussian Plume Model. With the new features, training data sets were calculated using the classification methods SVM and MLE. With these training data sets new sensor measurements may be assigned to the substances which are to be sought. The advantage of the presented method is that no feature reduction is needed and no loss of information occurs in the learning process.The classification results based on the new features have been compared with the classification based on a conventional method for feature extraction. It was proved that the recognition rate of the substances used with the new feature type is higher.The substance classification is primarily limited by the sensitivity of the semiconductor sensors, because sufficiently large sensor signals must have been provided to obtain appropriate substance patterns. At the present stage of development the method presented is suitable for the classification of substance groups, such as nitro aromatics or alcohols, but not for specific substances.     

Complete discriminant evaluation and feature extraction in kernel space for face recognition

This work proposes a method to decompose the kernel within-class eigenspace into two subspaces: a reliable subspace spanned mainly by the facial variation and an unreliable subspace due to limited number of training samples. A weighting function is proposed to circumvent undue scaling of eigenvectors corresponding to the unreliable small and zero eigenvalues. Eigenfeatures are then extracted by the discriminant evaluation in the whole kernel space. These efforts facilitate a discriminative and stable low-dimensional feature representation of the face image. Experimental results on FERET, ORL and GT databases show that our approach consistently outperforms other kernel based face recognition methods.

Fault diagnosis and classification framework using multi-scale classification based on kernel Fisher discriminant analysis for chemical process system

Fault detection and diagnosis (FDD) in chemical process systems is an important tool for effective process monitoring to ensure the safety of a process. Multi-scale classification offers various advantages for monitoring chemical processes generally driven by events in different time and frequency domains. However, there are issues when dealing with highly interrelated, complex, and noisy databases with large dimensionality. Therefore, a new method for the FDD framework is proposed based on wavelet analysis, kernel Fisher discriminant analysis (KFDA), and support vector machine (SVM) classifiers. The main objective of this work was to combine the advantages of these tools to enhance the performance of the diagnosis on a chemical process system. Initially, a discrete wavelet transform (DWT) was applied to extract the dynamics of the process at different scales. The wavelet coefficients obtained during the analysis were reconstructed using the inverse discrete wavelet transform (IDWT) method, which were then fed into the KFDA to produce discriminant vectors. Finally, the discriminant vectors were used as inputs for the SVM classification task. The SVM classifiers were utilized to classify the feature sets extracted by the proposed method. The performance of the proposed multi-scale KFDA-SVM method for fault classification and diagnosis was analysed and compared using a simulated Tennessee Eastman process as a benchmark. The results showed the improvements of the proposed multiscale KFDA-SVM framework with an average 96.79% of classification accuracy over the multi-scale KFDA-GMM (84.94%), and the established independent component analysis-SVM method (95.78%) of the faults in the Tennessee Eastman process.     

Feature extraction and dimensionality reduction by genetic programming based on the Fisher criterion

Abstract: Feature extraction helps to maximize the useful information within a feature vector, by reducing the dimensionality and making the classification effective and simple. In this paper, a novel feature extraction method is proposed: genetic programming (GP) is used to discover features, while the Fisher criterion is employed to assign fitness values. This produces non‐linear features for both two‐class and multiclass recognition, reflecting the discriminating information between classes. Compared with other GP‐based methods which need to generate c discriminant functions for solving c‐class (c>2) pattern recognition problems, only one single feature, obtained by a single GP run, appears to be highly satisfactory in this approach. The proposed method is experimentally compared with some non‐linear feature extraction methods, such as kernel generalized discriminant analysis and kernel principal component analysis. Results demonstrate the capability of the proposed approach to transform information from the high‐dimensional feature space into a single‐dimensional space by automatically discovering the relationships between data, producing improved performance.     

On the relation between discriminant analysis and mutual information for supervised linear feature extraction

This paper provides a unifying view of three discriminant linear feature extraction methods: linear discriminant analysis, heteroscedastic discriminant analysis and maximization of mutual information. We propose a model-independent reformulation of the criteria related to these three methods that stresses their similarities and elucidates their differences. Based on assumptions for the probability distribution of the classification data, we obtain sufficient conditions under which two or more of the above criteria coincide. It is shown that these conditions also suffice for Bayes optimality of the criteria. Our approach results in an information-theoretic derivation of linear discriminant analysis and heteroscedastic discriminant analysis. Finally, regarding linear discriminant analysis, we discuss its relation to multidimensional independent component analysis and derive suboptimality bounds based on information theory.     

基于小波变换的优化LDA人脸特征提取

运用小波进行图像分解提取低频子带图,并利用优化的线性判别分析(LDA)算法寻找最优投影子空间,从而映射提取人脸特征,实现人脸的分类识别。该方法避免了传统LDA算法中类内离散度矩阵非奇异的要求,解决了边缘类重叠问题,具有更广泛的应用空间。实验表明:该方法优于传统的LDA方法和主分量分析(PCA)方法。     

基于组合特征提取与多级SVM的轮胎花纹识别

基于轮胎花纹分类识别在交通与刑事部门的重要作用,提出了一种新的基于组合特征提取与多级SVM的轮胎花纹识别方法。分别采用非下采样Contourlet变换和灰度共生矩阵方法提取轮胎花纹特征;组合两种方法所提取的特征作为图像特征,并从中提取5个有效特征作为最终识别特征;运用提取的5个特征和多级支持向量机分类器完成轮胎花纹的分类识别。新的特征提取方法所得轮胎花纹特征分离度高,用决策树SVM分类器预测分类效果理想,对轮胎花纹的正确分类识别有着重要意义。     

一种LDA与SVM混合的多类分类方法

针对决策有向无环图支持向量机（ＤＤＡＧＳＶＭ）需训练大量支持向量机（ＳＶＭ）和误差积累的问题,提出一种线性判别分析（ＬＤＡ）与ＳＶＭ 混合的多类分类算法．首先根据高维样本在低维空间中投影的特点,给出一种优化ＬＤＡ 分类阈值;然后以优化ＬＤＡ 对每个二类问题的分类误差作为类间线性可分度,对线性可分度较低的问题采用非线性ＳＶＭ 加以解决,并以分类误差作为对应二类问题的可分度;最后将可分度作为混合ＤＤＡＧ 分类器的决策依据．实验表明,与ＤＤＡＧＳＶＭ 相比,所提出算法在确保泛化精度的条件下具有更高的训练和分类速度．     

局部KPLS特征提取的LSSVM软测量建模方法

针对复杂工业过程的非线性、变量间的强相关性以及工况时变的特点,提出了一种基于局部KPLS特征提取的LSSVM建模方法。该方法通过属性加权的欧式距离指标选取局部训练样本子集,利用KPLS算法对该子集进行特征提取,使用LSSVM算法在线建立局部软测量模型。实验结果表明,该方法可以有效实现特征提取,具有更好的推广能力和预测精度,比基于全局KPLS特征提取的LSSVM模型和未经特征提取的全局LSSVM模型具有更好的泛化能力。     

一种基于PCA的组合特征提取文本分类方法

为了获得更好的文本分类准确率和更快的执行效率, 研究了多种Web文本的特征提取方法, 通过对互信息(MI)、文档频率(DF)、信息增益(IG)和χ²统计（CHI）算法的研究, 利用其各自的优势互补, 提出一种基于主成分分析(PCA)的多重组合特征提取算法(PCA-CFEA)。通过PCA算法的正交变换快速地将文本特征空间降维, 再通过多重组合特征提取算法在降维后的特征空间中快速提取出更具代表性的特征项, 过滤掉一些代表性较弱的特征项, 最后使用SVM分类器对文本进行分类。实验结果表明, PCA-CFEA能有效地提高文本分类的正确率和执行效率。     

A new LDA-KL combined method for feature extraction and its generalisation

Linear discriminant analysis (LDA) is a well-known feature extraction technique. In this paper, we point out that LDA is not perfect because it only utilises the discriminatory information existing in the first-order statistical moments and ignores the information contained in the second-order statistical moments. We enhance LDA using the idea of a K-L expansion technique and develop a new LDA-KL combined method, which can make full use of both sections of discriminatory information. The proposed method is tested on the Concordia University CENPARMI handwritten numeral database. The experimental results indicate that the proposed LDA-KL method is more powerful than the existing techniques of LDA, K-L expansion and their combination: OLDA-PCA. What is more, the proposed method is further generalised to suit for feature extraction in the complex feature space and can be an effective tool for feature fusion.An erratum to this article can be found at     

基于支持向量机和正交设计的特征选择方法

正交设计利用较少的实验次数就可以找出因素间的最优搭配,支持向量机能处理小样本、具有很好的泛化能力且不受数据集维数的制约。结合二者的优势,提出了基于支持向量机和正交设计的特征选择方法,根据数据集的特征数目及相应正交表的结构,安排训练、测试,最后对优选出的特征子集检验,实验结果表明该特征选择方法能够去除冗余特征而且能取得比使用特征全集更高的分类率。     

基于KPLS的网络入侵特征抽取及检测方法

从特征抽取的角度研究提高入侵检测性能问题,提出应用核偏最小二乘(KPLS)进行入侵特征抽取和检测的方法.其优点在于KPLS能非线性地抽取输入特征的多个正交分量,并保持与输出类别的相关性,可同时完成入侵特征抽取和判别.将该方法应用于基于Linux主机的入侵检测实验,取得了比SVM和KPCR等方法更好的效果.     

A feature analysis for dimension reduction based on a data generation model with class factors and environment factors

Currently, high-dimensional data such as image data is widely used in the domain of pattern classification and signal processing. When using high-dimensional data, feature analysis methods such as PCA (principal component analysis) and LDA (linear discriminant analysis) are usually required in order to reduce memory usage or computational complexity as well as to increase classification performance. We propose a feature analysis method for dimension reduction based on a data generation model that is composed of two types of factors: class factors and environment factors. The class factors, which are prototypes of the classes, contain important information required for discriminating between various classes. The environment factors, which represent distortions of the class prototypes, need to be diminished for obtaining high class separability. Using the data generation model, we aimed to exclude environment factors and extract low-dimensional class factors from the original data. By performing computational experiments on artificial data sets and real facial data sets, we confirmed that the proposed method can efficiently extract low-dimensional features required for classification and has a better performance than the conventional methods.     

基于核非负分解的人脸图像特征提取与分类

针对非负矩阵分解算法在样本维数过高情况下收敛效果差的问题,提出了一种核矩阵非负分解算法。通过核映射方法获得表征样本间相似度的核矩阵,以减小样本类内散度,增大样本类间散度,从而改善样本内部噪声干扰,提高样本间的线性可分度;再将核矩阵在非负条件约束下分解为基向量及其加权系数矩阵,用系数矩阵作为原样本特征。经人脸图像特征提取与分类实验验证,新算法可更好地提取高维人脸图像的低维特征,提高分类正确率。     

苹果近红外光谱的特征提取和分类研究

采用近红外光谱分析法对不同种类的苹果样品进行分类,提出一种基于非相关判别转换的苹果近红外光谱定性分析新方法。实验分别采用主成分分析、Fisher判别分析和非相关判别转换三种方法对苹果光谱数据进行特征提取,并使用K-近邻分类算法建立三种苹果分类识别模型,最后使用"留一"交叉验证法进行模型检验。结果表明,使用非相关判别转换方法建立的模型正确识别率优于使用主成分分析和Fisher判别分析建立的模型。     

基于多子空间线性判别分析的特征提取与分类

为了解决高维小样本数据的分类中Fisherface思想判别分析方法的不足,在最大散度差准则的基础上,提出了利用多线性子空间技术对每类样本进行单独描述的方法,该方法能更准确地反映样本在类内类间的分布关系.在分类中不是以距离作为判别依据,而是按照贝叶斯决策规则得到的隶属置信度作为衡量标准.实验结果表明了该方法的有效性,和同类方法相比,有更高的识别率.