Graph embedding and extensions: a general framework for dimensionality reduction

A large family of algorithms - supervised or unsupervised; stemming from statistics or geometry theory - has been designed to provide different solutions to the problem of dimensionality reduction. Despite the different motivations of these algorithms, we present in this paper a general formulation known as graph embedding to unify them within a common framework. In graph embedding, each algorithm can be considered as the direct graph embedding or its linear/kernel/tensor extension of a specific intrinsic graph that describes certain desired statistical or geometric properties of a data set, with constraints from scale normalization or a penalty graph that characterizes a statistical or geometric property that should be avoided. Furthermore, the graph embedding framework can be used as a general platform for developing new dimensionality reduction algorithms. By utilizing this framework as a tool, we propose a new supervised dimensionality reduction algorithm called marginal Fisher analysis in which the intrinsic graph characterizes the intraclass compactness and connects each data point with its neighboring points of the same class, while the penalty graph connects the marginal points and characterizes the interclass separability. We show that MFA effectively overcomes the limitations of the traditional linear discriminant analysis algorithm due to data distribution assumptions and available projection directions. Real face recognition experiments show the superiority of our proposed MFA in comparison to LDA, also for corresponding kernel and tensor extensions     

Locality preserving and global discriminant projection with prior information

Existing supervised and semi-supervised dimensionality reduction methods utilize training data only with class labels being associated to the data samples for classification. In this paper, we present a new algorithm called locality preserving and global discriminant projection with prior information (LPGDP) for dimensionality reduction and classification, by considering both the manifold structure and the prior information, where the prior information includes not only the class label but also the misclassification of marginal samples. In the LPGDP algorithm, the overlap among the class-specific manifolds is discriminated by a global class graph, and a locality preserving criterion is employed to obtain the projections that best preserve the within-class local structures. The feasibility of the LPGDP algorithm has been evaluated in face recognition, object categorization and handwritten Chinese character recognition experiments. Experiment results show the superior performance of data modeling and classification to other techniques, such as linear discriminant analysis, locality preserving projection, discriminant locality preserving projection and marginal Fisher analysis.     

Effective multiplicative updates for non-negative discriminative learning in multimodal dimensionality reduction

Fisher discriminant analysis gives the unsatisfactory results if points in the same class have within-class multimodality and fails to produce the non-negativity of projection vectors. In this paper, we focus on the newly formulated within and between-class scatters based supervised locality preserving dimensionality reduction problem and propose an effective dimensionality reduction algorithm, namely, Multiplicative Updates based non-negative Discriminative Learning (MUNDL), which optimally seeks to obtain two non-negative embedding transformations with high preservation and discrimination powers for two data sets in different classes such that nearby sample pairs in the original space compact in the learned embedding space, under which the projections of the original data in different classes can be appropriately separated from each other. We also show that MUNDL can be easily extended to nonlinear dimensionality reduction scenarios by employing the standard kernel trick. We verify the feasibility and effectiveness of MUNDL by conducting extensive data visualization and classification experiments. Numerical results on some benchmark UCI and real-world datasets show the MUNDL method tends to capture the intrinsic local and multimodal structure characteristics of the given data and outperforms some established dimensionality reduction methods, while being much more efficient.     

Trace ratio criterion based generalized discriminative learning for semi-supervised dimensionality reduction

Dealing with high-dimensional data has always been a major problem in many pattern recognition and machine learning applications. Trace ratio criterion is a criterion that can be applicable to many dimensionality reduction methods as it directly reflects Euclidean distance between data points of within or between classes. In this paper, we analyze the trace ratio problem and propose a new efficient algorithm to find the optimal solution. Based on the proposed algorithm, we are able to derive an orthogonal constrained semi-supervised learning framework. The new algorithm incorporates unlabeled data into training procedure so that it is able to preserve the discriminative structure as well as geometrical structure embedded in the original dataset. Under such a framework, many existing semi-supervised dimensionality reduction methods such as SDA, Lap-LDA, SSDR, SSMMC, can be improved using our proposed framework, which can also be used to formulate a corresponding kernel framework for handling nonlinear problems. Theoretical analysis indicates that there are certain relationships between linear and nonlinear methods. Finally, extensive simulations on synthetic dataset and real world dataset are presented to show the effectiveness of our algorithms. The results demonstrate that our proposed algorithm can achieve great superiority to other state-of-art algorithms.     

Adaptive nonlinear manifolds and their applications to pattern recognition

Dimensionality reduction has long been associated with retinotopic mapping for understanding cortical maps. Multisensory information is processed, fused and mapped to an essentially 2-D cortex in an information preserving manner. Data processing and projection techniques inspired by this biological mechanism are playing an increasingly important role in pattern recognition, computational intelligence, data mining, information retrieval and image recognition. Dimensionality reduction involves reduction of features or volume of data and has become an essential step of information processing in many fields. The topic of manifold learning has recently attracted a great deal of attention, and a number of advanced techniques for extracting nonlinear manifolds and reducing data dimensions have been proposed from statistics, geometry theory and adaptive neural networks. This paper provides an overview of this challenging and emerging topic and discusses various recent methods such as self-organizing map (SOM), kernel PCA, principal manifold, isomap, local linear embedding, and Laplacian eigenmap. Many of them can be considered in a learning manifold framework. The paper further elaborates on the biologically inspired SOM model and its metric preserving variant ViSOM under the framework of adaptive manifold; and their applications in dimensionality reduction with face recognition are investigated. The experiments demonstrate that adaptive ViSOM-based methods produce markedly improved performance over the others due to their metric scaling and preserving properties along the nonlinear manifold.     

Nonlinear process monitoring based on kernel global–local preserving projections

A new nonlinear dimensionality reduction method called kernel global–local preserving projections (KGLPP) is developed and applied for fault detection. KGLPP has the advantage of preserving global and local data structures simultaneously. The kernel principal component analysis (KPCA), which only preserves the global Euclidean structure of data, and the kernel locality preserving projections (KLPP), which only preserves the local neighborhood structure of data, are unified in the KGLPP framework. KPCA and KLPP can be easily derived from KGLPP by choosing some particular values of parameters. As a result, KGLPP is more powerful than KPCA and KLPP in capturing useful data characteristics. A KGLPP-based monitoring method is proposed for nonlinear processes. T² and SPE statistics are constructed in the feature space for fault detection. Case studies in a nonlinear system and in the Tennessee Eastman process demonstrate that the KGLPP-based method significantly outperforms KPCA, KLPP and GLPP-based methods, in terms of higher fault detection rates and better fault sensitivity.     

A unified dimensionality reduction framework for semi-paired and semi-supervised multi-view data

Canonical correlation analysis (CCA) is a popular and powerful dimensionality reduction method to analyze paired multi-view data. However, when facing semi-paired and semi-supervised multi-view data which widely exist in real-world problems, CCA usually performs poorly due to its requirement of data pairing between different views and un-supervision in nature. Recently, several extensions of CCA have been proposed, however, they just handle the semi-paired scenario by utilizing structure information in each view or just deal with semi-supervised scenario by incorporating the discriminant information. In this paper, we present a general dimensionality reduction framework for semi-paired and semi-supervised multi-view data which naturally generalizes existing related works by using different kinds of prior information. Based on the framework, we develop a novel dimensionality reduction method, termed as semi-paired and semi-supervised generalized correlation analysis (S²GCA). S²GCA exploits a small amount of paired data to perform CCA and at the same time, utilizes both the global structural information captured from the unlabeled data and the local discriminative information captured from the limited labeled data to compensate the limited pairedness. Consequently, S²GCA can find the directions which make not only maximal correlation between the paired data but also maximal separability of the labeled data. Experimental results on artificial and four real-world datasets show its effectiveness compared to the existing related dimensionality reduction methods.     

Dimensionality reduction via kernel sparse representation

Dimensionality reduction (DR) methods based on sparse representation as one of the hottest research topics have achieved remarkable performance in many applications in recent years. However, it’s a challenge for existing sparse representation based methods to solve nonlinear problem due to the limitations of seeking sparse representation of data in the original space. Motivated by kernel tricks, we proposed a new framework called empirical kernel sparse representation (EKSR) to solve nonlinear problem. In this framework, nonlinear separable data are mapped into kernel space in which the nonlinear similarity can be captured, and then the data in kernel space is reconstructed by sparse representation to preserve the sparse structure, which is obtained by minimizing a ?1 regularization-related objective function. EKSR provides new insights into dimensionality reduction and extends two models: 1) empirical kernel sparsity preserving projection (EKSPP), which is a feature extraction method based on sparsity preserving projection (SPP); 2) empirical kernel sparsity score (EKSS), which is a feature selection method based on sparsity score (SS). Both of the two methods can choose neighborhood automatically as the natural discriminative power of sparse representation. Compared with several existing approaches, the proposed framework can reduce computational complexity and be more convenient in practice.     

Matrix-variate and higher-order probabilistic projections

Feature extraction from two-dimensional or higher-order data, such as face images and surveillance videos, have recently been an active research area. There have been several 2D or higher-order PCA-style dimensionality reduction algorithms, but they mostly lack probabilistic interpretations and are difficult to apply with, e.g., incomplete data. It is also hard to extend these algorithms for applications where a certain region of the data point needs special focus in the dimensionality reduction process (e.g., the facial region in a face image). In this paper we propose a probabilistic dimensionality reduction framework for 2D and higher-order data. It specifies a particular generative process for this type of data, and leads to better understanding of some 2D and higher-order PCA-style algorithms. In particular, we show it actually takes several existing algorithms as its (non-probabilistic) special cases. We develop efficient iterative learning algorithms within this framework and study the theoretical properties of the stationary points. The model can be easily extended to handle special regions in the high-order data. Empirical studies on several benchmark data and real-world cardiac ultrasound images demonstrate the strength of this framework.     

基于稀疏和近邻保持的极限学习机降维

近邻与稀疏保持投影已被广泛应用于降维方法,通过优化得到满足近邻结构或稀疏结构的降维投影矩阵,然而这类方法多数只考虑单一结构特征.此外,多数非线性降维方法无法求出显式的映射函数,极大地限制了降维方法的应用.为克服这些问题,本文借鉴极限学习机的思想,提出面向聚类的基于稀疏和近邻保持的极限学习机降维算法（SNP-ELM）.SNP-ELM算法是一种非线性无监督降维方法,在降维过程中同时考虑数据的稀疏结构与近邻结构.在人造数据、Wine数据和6个基因表达数据上进行实验,实验结果表明该算法优于其他降维方法.     

Local sparse representation projections for face recognition

How to define the sparse affinity weight matrices is still an open problem in existing manifold learning algorithm. In this paper, we propose a novel supervised learning method called local sparse representation projections (LSRP) for linear dimensionality reduction. Differing from sparsity preserving projections (SPP) and the recent manifold learning methods such as locality preserving projections (LPP), LSRP introduces the local sparse representation information into the objective function. Although there are no labels used in the local sparse representation, it still can provide better measure coefficients and significant discriminant abilities. By combining the local interclass neighborhood relationships and sparse representation information, LSRP aims to preserve the local sparse reconstructive relationships of the data and simultaneously maximize the interclass separability. Comprehensive comparison and extensive experiments show that LSRP achieves higher recognition rates than principle component analysis, linear discriminant analysis and the state-of-the-art techniques such as LPP, SPP and maximum variance projections.     

Locally Minimizing Embedding and Globally Maximizing Variance: Unsupervised Linear Difference Projection for Dimensionality Reduction

Recently, many dimensionality reduction algorithms, including local methods and global methods, have been presented. The representative local linear methods are locally linear embedding (LLE) and linear preserving projections (LPP), which seek to find an embedding space that preserves local information to explore the intrinsic characteristics of high dimensional data. However, both of them still fail to nicely deal with the sparsely sampled or noise contaminated datasets, where the local neighborhood structure is critically distorted. On the contrary, principal component analysis (PCA), the most frequently used global method, preserves the total variance by maximizing the trace of feature variance matrix. But PCA cannot preserve local information due to pursuing maximal variance. In order to integrate the locality and globality together and avoid the drawback in LLE and PCA, in this paper, inspired by the dimensionality reduction methods of LLE and PCA, we propose a new dimensionality reduction method for face recognition, namely, unsupervised linear difference projection (ULDP). This approach can be regarded as the integration of a local approach (LLE) and a global approach (PCA), so that it has better performance and robustness in applications. Experimental results on the ORL, YALE and AR face databases show the effectiveness of the proposed method on face recognition.     

稀疏局部Fisher判别分析

提出一种稀疏局部Fisher判别分析（Sparsity Local Fisher Discriminant Analysis,SLFDA）。该算法在局部Fisher判别分析降维的基础上,通过平衡参数引入稀疏保持投影,在投影降维过程中保持了数据的全局几何结构和局部近邻信息。在UCI数据集和YaleB人脸数据集上的实验表明,该算法融合局部Fisher判别分析和稀疏保持投影的优点;与现有的半监督局部Fisher判别分析降维算法相比,该算法提高了基于最短欧氏距离的分类算法的精度。     

Maximal local interclass embedding with application to face recognition

Dimensionality reduction of high dimensional data is involved in many problems in information processing. A new dimensionality reduction approach called maximal local interclass embedding (MLIE) is developed in this paper. MLIE can be viewed as a linear approach of a multimanifolds-based learning framework, in which the information of neighborhood is integrated with the local interclass relationships. In MLIE, the local interclass graph and the intrinsic graph are constructed to find a set of projections that maximize the local interclass scatter and the local intraclass compactness simultaneously. This characteristic makes MLIE more powerful than marginal Fisher analysis (MFA). MLIE maintains all the advantages of MFA. Moreover, the computational complexity of MLIE is less than that of MFA. The proposed algorithm is applied to face recognition. Experiments have been performed on the Yale, AR and ORL face image databases. The experimental results show that owing to the locally discriminating property, MLIE consistently outperforms up-to-date MFA, Smooth MFA, neighborhood preserving embedding and locality preserving projection in face recognition.     

Dynamic transition embedding for image feature extraction and recognition

In this paper, we propose a novel method called dynamic transition embedding (DTE) for linear dimensionality reduction. Differing from the recently proposed manifold learning-based methods, DTE introduces the dynamic transition information into the objective function by characterizing the Markov transition processes of the data set in time t(t?>?0). In the DTE framework, running the Markov chain forward in time, or equivalently, taking the larger powers of Markov transition matrices integrates the local geometry and, therefore, reveals relevant geometric structures of the data set at different timescales. Since the Markov transition matrices defined by the connectivity on a graph contain the intrinsic geometry information of the data points, the elements of the Markov transition matrices can be viewed as the probabilities or the similarities between two points. Thus, minimizing the errors of the probability reconstruction or similarity reconstruction instead of the least-square reconstruction in the well-known manifold learning algorithms will obtain the optimal linear projections with respect to preserving the intrinsic Markov processes of the data set. Comprehensive comparisons and extensive experiments show that DTE achieves higher recognition rates than some well-known linear dimensionality reduction techniques.     

多标记学习自编码网络无监督维数约简

多标记学习是针对一个实例同时与一组标签相关联而提出的一种机器学习框架,是该领域研究热点之一,降维是多标记学习一个重要且具有挑战性的工作。针对有监督的多标记维数约简方法,提出一种无监督自编码网络的多标记降维方法。首先,通过构建自编码神经网络,对输入数据进行编码和解码输出;然后,引入稀疏约束计算总体成本,使用梯度下降法进行迭代求解;最后,通过深度学习训练获得自编码网络学习模型,提取数据特征实现维数约简。实验中使用多标记算法ML-kNN做分类器,在6个公开数据集上与其他4种方法对比。实验结果表明,该方法能够在不使用标记的情况下有效提取特征,降低多标记数据维度,稳定提高多标记学习性能。     

Locality-preserved maximum information projection.

Dimensionality reduction is usually involved in the domains of artificial intelligence and machine learning. Linear projection of features is of particular interest for dimensionality reduction since it is simple to calculate and analytically analyze. In this paper, we propose an essentially linear projection technique, called locality-preserved maximum information projection (LPMIP), to identify the underlying manifold structure of a data set. LPMIP considers both the within-locality and the between-locality in the processing of manifold learning. Equivalently, the goal of LPMIP is to preserve the local structure while maximize the out-of-locality (global) information of the samples simultaneously. Different from principal component analysis (PCA) that aims to preserve the global information and locality-preserving projections (LPPs) that is in favor of preserving the local structure of the data set, LPMIP seeks a tradeoff between the global and local structures, which is adjusted by a parameter alpha, so as to find a subspace that detects the intrinsic manifold structure for classification tasks. Computationally, by constructing the adjacency matrix, LPMIP is formulated as an eigenvalue problem. LPMIP yields orthogonal basis functions, and completely avoids the singularity problem as it exists in LPP. Further, we develop an efficient and stable LPMIP/QR algorithm for implementing LPMIP, especially, on high-dimensional data set. Theoretical analysis shows that conventional linear projection methods such as (weighted) PCA, maximum margin criterion (MMC), linear discriminant analysis (LDA), and LPP could be derived from the LPMIP framework by setting different graph models and constraints. Extensive experiments on face, digit, and facial expression recognition show the effectiveness of the proposed LPMIP method.     

Patch Alignment for Dimensionality Reduction

Spectral analysis-based dimensionality reduction algorithms are important and have been popularly applied in data mining and computer vision applications. To date many algorithms have been developed, e.g., principal component analysis, locally linear embedding, Laplacian eigenmaps, and local tangent space alignment. All of these algorithms have been designed intuitively and pragmatically, i.e., on the basis of the experience and knowledge of experts for their own purposes. Therefore, it will be more informative to provide a systematic framework for understanding the common properties and intrinsic difference in different algorithms. In this paper, we propose such a framework, named "patch alignment,” which consists of two stages: part optimization and whole alignment. The framework reveals that 1) algorithms are intrinsically different in the patch optimization stage and 2) all algorithms share an almost identical whole alignment stage. As an application of this framework, we develop a new dimensionality reduction algorithm, termed Discriminative Locality Alignment (DLA), by imposing discriminative information in the part optimization stage. DLA can 1) attack the distribution nonlinearity of measurements; 2) preserve the discriminative ability; and 3) avoid the small-sample-size problem. Thorough empirical studies demonstrate the effectiveness of DLA compared with representative dimensionality reduction algorithms.     

中心近邻嵌入学习算法的人脸识别研究

针对人脸识别问题,提出了一种中心近邻嵌入的学习算法,其与经典的局部线性嵌入和保局映射不同,它是一种有监督的线性降维方法。该方法首先通过计算各类样本中心,并引入中心近邻距离代替两样本点之间的直接距离作为权系数函数的输入;然后再保持中心近邻的几何结构不变的情况下把高维数据嵌入到低维坐标系中。通过中心近邻嵌入学习算法与其他3种人脸识别方法(即主成分分析、线形判别分析及保局映射)在ORL、Yale及UMIST人脸库上进行的比较实验结果表明,它在高维数据低维可视化和人脸识别效果等方面均较其他3种方法取得了更好的效果。     

Adaptive learning codebook for action recognition

Learning a compact and yet discriminative codebook is an important procedure for local feature-based action recognition. A common procedure involves two independent phases: reducing the dimensionality of local features and then performing clustering. Since the two phases are disconnected, dimensionality reduction does not necessarily capture the dimensions that are greatly helpful for codebook creation. What’s more, some dimensionality reduction techniques such as the principal component analysis do not take class separability into account and thus may not help build an effective codebook. In this paper, we propose the weighted adaptive metric learning (WAML) which integrates the two independent phases into a unified optimization framework. This framework enables to select indispensable and crucial dimensions for building a discriminative codebook. The dimensionality reduction phase in the WAML is optimized for class separability and adaptively adjusts the distance metric to improve the separability of data. In addition, the video word weighting is smoothly incorporated into the WAML to accurately generate video words. Experimental results demonstrate that our approach builds a highly discriminative codebook and achieves comparable results to other state-of-the-art approaches.