KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition

This paper examines the theory of kernel Fisher discriminant analysis (KFD) in a Hilbert space and develops a two-phase KFD framework, i.e., kernel principal component analysis (KPCA) plus Fisher linear discriminant analysis (LDA). This framework provides novel insights into the nature of KFD. Based on this framework, the authors propose a complete kernel Fisher discriminant analysis (CKFD) algorithm. CKFD can be used to carry out discriminant analysis in "double discriminant subspaces." The fact that, it can make full use of two kinds of discriminant information, regular and irregular, makes CKFD a more powerful discriminator. The proposed algorithm was tested and evaluated using the FERET face database and the CENPARMI handwritten numeral database. The experimental results show that CKFD outperforms other KFD algorithms.     

Large-scale maximum margin discriminant analysis using core vector machines.

Large-margin methods, such as support vector machines (SVMs), have been very successful in classification problems. Recently, maximum margin discriminant analysis (MMDA) was proposed that extends the large-margin idea to feature extraction. It often outperforms traditional methods such as kernel principal component analysis (KPCA) and kernel Fisher discriminant analysis (KFD). However, as in the SVM, its time complexity is cubic in the number of training points m, and is thus computationally inefficient on massive data sets. In this paper, we propose an (1+epsilon)(2)-approximation algorithm for obtaining the MMDA features by extending the core vector machine. The resultant time complexity is only linear in m, while its space complexity is independent of m. Extensive comparisons with the original MMDA, KPCA, and KFD on a number of large data sets show that the proposed feature extractor can improve classification accuracy, and is also faster than these kernel-based methods by over an order of magnitude.     

核学习机研究

该文概述了近年来机器学习研究领域的一个热点问题———核学习机。首先分析了核方法的主要思想,然后着重介绍了几种新近发展的核学习机,包括支持向量机、核的Fisher判别分析等有监督学习算法及核的主分量分析等无监督学习算法,最后讨论了其应用及前景展望。     

Learning the kernel matrix by maximizing a KFD-based class separability criterion

The advantage of a kernel method often depends critically on a proper choice of the kernel function. A promising approach is to learn the kernel from data automatically. In this paper, we propose a novel method for learning the kernel matrix based on maximizing a class separability criterion that is similar to those used by linear discriminant analysis (LDA) and kernel Fisher discriminant (KFD). It is interesting to note that optimizing this criterion function does not require inverting the possibly singular within-class scatter matrix which is a computational problem encountered by many LDA and KFD methods. We have conducted experiments on both synthetic data and real-world data from UCI and FERET, showing that our method consistently outperforms some previous kernel learning methods.     

A novel neural-network approach of analog fault diagnosis based on kernel discriminant analysis and particle swarm optimization

Kernel principal component analysis (KPCA) and kernel linear discriminant analysis (KLDA) are two commonly used and effective methods for dimensionality reduction and feature extraction. In this paper, we propose a KLDA method based on maximal class separability for extracting the optimal features of analog fault data sets, where the proposed KLDA method is compared with principal component analysis (PCA), linear discriminant analysis (LDA) and KPCA methods. Meanwhile, a novel particle swarm optimization (PSO) based algorithm is developed to tune parameters and structures of neural networks jointly. Our study shows that KLDA is overall superior to PCA, LDA and KPCA in feature extraction performance and the proposed PSO-based algorithm has the properties of convenience of implementation and better training performance than Back-propagation algorithm. The simulation results demonstrate the effectiveness of these methods.     

一种新的核线性鉴别分析算法及其在人脸识别上的应用

基于核策略的核Fisher鉴别分析（KFD）算法已成为非线性特征抽取的最有效方法之一。但是先前的基于核Fisher鉴别分析算法的特征抽取过程都是基于2值分类问题而言的。如何从重叠（离群）样本中抽取有效的分类特征没有得到有效的解决。本文在结合模糊集理论的基础上,利用模糊隶属度函数的概念,在特征提取过程中融入了样本的分布信息,提出了一种新的核Fisher鉴别分析方法——模糊核鉴别分析算法。在ORL人脸数据库上的实验结果验证了该算法的有效性。     

融合FDA-PCMC样本分类的KPCA故障检测新算法

针对处理实际工业过程中提取的建模样本不纯而导致故障检测失效的问题,提出一种新的融合Fisher判别分析-可能性C-均值聚类(FDA-PCMC)的核主元分析(KPCA)故障检测算法.通过FDA特征提取、初分类和PCMC聚类相结合的方代来实现建模样本的有效分类和提纯,然后使用KPCA进行实时故障检测.对Tennessee Eastman(TE)过程的仿真研宄结果表明了该算法的可行性和有效性.     

Choosing parameters of kernel subspace LDA for recognition of face images under pose and illumination variations.

This paper addresses the problem of automatically tuning multiple kernel parameters for the kernel-based linear discriminant analysis (LDA) method. The kernel approach has been proposed to solve face recognition problems under complex distribution by mapping the input space to a high-dimensional feature space. Some recognition algorithms such as the kernel principal components analysis, kernel Fisher discriminant, generalized discriminant analysis, and kernel direct LDA have been developed in the last five years. The experimental results show that the kernel-based method is a good and feasible approach to tackle the pose and illumination variations. One of the crucial factors in the kernel approach is the selection of kernel parameters, which highly affects the generalization capability and stability of the kernel-based learning methods. In view of this, we propose an eigenvalue-stability-bounded margin maximization (ESBMM) algorithm to automatically tune the multiple parameters of the Gaussian radial basis function kernel for the kernel subspace LDA (KSLDA) method, which is developed based on our previously developed subspace LDA method. The ESBMM algorithm improves the generalization capability of the kernel-based LDA method by maximizing the margin maximization criterion while maintaining the eigenvalue stability of the kernel-based LDA method. An in-depth investigation on the generalization performance on pose and illumination dimensions is performed using the YaleB and CMU PIE databases. The FERET database is also used for benchmark evaluation. Compared with the existing PCA-based and LDA-based methods, our proposed KSLDA method, with the ESBMM kernel parameter estimation algorithm, gives superior performance.     

核函数在主成分分析中的应用

主成分分析在对线性数据进行降维时非常有效,核函数能够将线性不可分的数据映射到高维希尔伯特空间中可能可分。将核函数应用到主成分分析中成为核主成分分析。从核函数的性质、核函数的参数调整、核函数的构造等方面对核主成分分析进行应用与实现,并结合核Fisher判别分析,对样例数据进行核主成分分析,结论表明,效果良好,但执行速度较慢,需要后续改进。     

基于核Fisher判别和加权码书映射的音频信号削波修复方法

本文基于核Fisher判别（Kernel Fisher Discriminant, KFD）和加权码书映射（Weighted Codebook Mapping, WCBM）,提出了一种MDCT（Modified Discrete Cosine Transform）域的音频信号削波修复方法。首先根据音频信号的MDCT系数提取子带包络等四种削波特征参数;其次,利用这些特征参数训练检测音频信号出现削波的核Fisher分类器;最后,利用子带包络的WCBM来修复音频信号的削波。测试结果表明,本文所提方法能有效修复音频信号的削波,其性能优于现有的几种削波修复方法。     

人脸识别中基于核的子空间鉴别分析

尽管基于Fisher准则的线性鉴别分析被公认为特征抽取的有效方法之一，并被成功地用于人脸识别，但是由于光照变化、人脸表情和姿势变化，实际上的人脸图像分布是十分复杂的，因此，抽取非线性鉴别特征显得十分必要。为了能利用非线性鉴别特征进行人脸识别，提出了一种基于核的子空间鉴别分析方法。该方法首先利用核函数技术将原始样本隐式地映射到高维（甚至无穷维）特征空间；然后在高维特征空间里，利用再生核理论来建立基于广义Fisher准则的两个等价模型；最后利用正交补空间方法求得最优鉴别矢量来进行人脸识别。在ORL和NUST603两个人脸数据库上，对该方法进行了鉴别性能实验，得到了识别率分别为94％和99．58％的实验结果，这表明该方法与核组合方法的识别结果相当，且明显优于KPCA和Kernel fisherfaces方法的识别结果。     

基于规范化KDDA的人脸识别算法

传统的PCA和LDA算法受限于“小样本问题”,且对像素的高阶相关性不敏感。论文将核函数方法与规范化LDA相结合,将原图像空间通过非线性映射变换到高维特征空间,并借助于“核技巧”在新的空间中应用鉴别分析方法。通过对ORL人脸库的大量实验表明,该方法在特征提取方面优于PCA,KPCA,LDA等其他方法,在简化分类器的同时,也可以获得高识别率。     

改进的PCA-LDA人脸识别算法的研究

主成分分析算法(PCA)和线性鉴别分析算法(LDA)被广泛用于人脸识别技术中,但是PCA由于其计算复杂度高,致使人脸识别的实时性达不到要求.线性鉴别分析算法存在"小样本"和"边缘类"问题,降低了人脸识别的准确性.针对上述问题,提出使用二维主成分分析法(2DPCA)与改进的线性鉴别分析法相融合的方法.二维主成分分析法提取...     

自适应模糊支持向量机算法研究

一个有效的核方法通常取决于选择一个合适的核函数。目前研究核方法的热点是从数据中自动地进行核学习。提出基于最优分类标准的核学习方法,这个标准类似于线性鉴别分析和核Fisher判别式。并把此算法应用于模糊支持向量机多类分类器设计上,在ORL人脸数据集和Iris数据集上的实验验证了该算法的可行性。     

Kernel-based nonlinear discriminant analysis for face recognition

Linear subspace analysis methods have been successfully applied to extract features for face recognition.But they are inadequate to represent the complex and nonlinear variations of real face images,such as illumination,facial expression and pose variations,because of their linear properties.In this paper,a nonlinear subspace analysis method,Kernel-based Nonlinear Discriminant Analysis (KNDA),is presented for face recognition,which combines the nonlinear kernel trick with the linear subspace analysis method-Fisher Linear Discriminant Analysis (FLDA).First,the kernel trick is used to project the input data into an implicit feature space,then FLDA is performed in this feature space.Thus nonlinear discriminant features of the input data are yielded.In addition,in order to reduce the computational complexity,a geometry-based feature vectors selection scheme is adopted.Another similar nonlinear subspace analysis is Kernel-based Principal Component Analysis (KPCA),which combines the kernel trick with linear Principal Component Analysis (PCA).Experiments are performed with the polynomial kernel,and KNDA is compared with KPCA and FLDA.Extensive experimental results show that KNDA can give a higher recognition rate than KPCA and FLDA.     

Feature scaling for kernel fisher discriminant analysis using leave-one-out cross validation

Kernel fisher discriminant analysis (KFD) is a successful approach to classification. It is well known that the key challenge in KFD lies in the selection of free parameters such as kernel parameters and regularization parameters. Here we focus on the feature-scaling kernel where each feature individually associates with a scaling factor. A novel algorithm, named FS-KFD, is developed to tune the scaling factors and regularization parameters for the feature-scaling kernel. The proposed algorithm is based on optimizing the smooth leave-one-out error via a gradient-descent method and has been demonstrated to be computationally feasible. FS-KFD is motivated by the following two fundamental facts: the leave-one-out error of KFD can be expressed in closed form and the step function can be approximated by a sigmoid function. Empirical comparisons on artificial and benchmark data sets suggest that FS-KFD improves KFD in terms of classification accuracy.     

Face recognition using kernel direct discriminant analysis algorithms

Techniques that can introduce low-dimensional feature representation with enhanced discriminatory power is of paramount importance in face recognition (FR) systems. It is well known that the distribution of face images, under a perceivable variation in viewpoint, illumination or facial expression, is highly nonlinear and complex. It is, therefore, not surprising that linear techniques, such as those based on principle component analysis (PCA) or linear discriminant analysis (LDA), cannot provide reliable and robust solutions to those FR problems with complex face variations. In this paper, we propose a kernel machine-based discriminant analysis method, which deals with the nonlinearity of the face patterns' distribution. The proposed method also effectively solves the so-called "small sample size" (SSS) problem, which exists in most FR tasks. The new algorithm has been tested, in terms of classification error rate performance, on the multiview UMIST face database. Results indicate that the proposed methodology is able to achieve excellent performance with only a very small set of features being used, and its error rate is approximately 34% and 48% of those of two other commonly used kernel FR approaches, the kernel-PCA (KPCA) and the generalized discriminant analysis (GDA), respectively.     

基于规范化KDDA的人脸识别

传统的PCA和LDA算法受限于“小样本问题”,且对象素的高阶相关性不敏感。本文将核函数方法与规范化LDA相结合,将原图像空间通过非线性映射变换到高维特征空间,并借助于“核技巧”在新的空间中应用鉴别分析方法。通过对ORL人脸库的大量实验研究表明,本文方法在特征提取方面明显优于PCA,KPCA,LDA等其他传统的人脸识别方法,在简化分类器的同时,也可以获得高识别率。     

Perturbation LDA: Learning the difference between the class empirical mean and its expectation

Fisher's linear discriminant analysis (LDA) is popular for dimension reduction and extraction of discriminant features in many pattern recognition applications, especially biometric learning. In deriving the Fisher's LDA formulation, there is an assumption that the class empirical mean is equal to its expectation. However, this assumption may not be valid in practice. In this paper, from the “perturbation” perspective, we develop a new algorithm, called perturbation LDA (P-LDA), in which perturbation random vectors are introduced to learn the effect of the difference between the class empirical mean and its expectation in Fisher criterion. This perturbation learning in Fisher criterion would yield new forms of within-class and between-class covariance matrices integrated with some perturbation factors. Moreover, a method is proposed for estimation of the covariance matrices of perturbation random vectors for practical implementation. The proposed P-LDA is evaluated on both synthetic data sets and real face image data sets. Experimental results show that P-LDA outperforms the popular Fisher's LDA-based algorithms in the undersampled case.     

Robust kernel discriminant analysis using fuzzy memberships

Linear discriminant analysis (LDA) is a simple but widely used algorithm in the area of pattern recognition. However, it has some shortcomings in that it is sensitive to outliers and limited to linearly separable cases. To solve these problems, in this paper, a non-linear robust variant of LDA, called robust kernel fuzzy discriminant analysis (RKFDA) is proposed. RKFDA uses fuzzy memberships to reduce the effect of outliers and adopts kernel methods to accommodate non-linearly separable cases. There have been other attempts to solve the problems of LDA, including attempts using kernels. However, RKFDA, encompassing previous methods, is the most general one. Furthermore, theoretical analysis and experimental results show that RKFDA is superior to other existing methods in solving the problems.