Stable local dimensionality reduction approaches

Dimensionality reduction is a big challenge in many areas. A large number of local approaches, stemming from statistics or geometry, have been developed. However, in practice these local approaches are often in lack of robustness, since in contrast to maximum variance unfolding (MVU), which explicitly unfolds the manifold, they merely characterize local geometry structure. Moreover, the eigenproblems that they encounter, are hard to solve. We propose a unified framework that explicitly unfolds the manifold and reformulate local approaches as the semi-definite programs instead of the above-mentioned eigenproblems. Three well-known algorithms, locally linear embedding (LLE), laplacian eigenmaps (LE) and local tangent space alignment (LTSA) are reinterpreted and improved within this framework. Several experiments are presented to demonstrate the potential of our framework and the improvements of these local algorithms.     

Trace ratio criterion based generalized discriminative learning for semi-supervised dimensionality reduction

Dealing with high-dimensional data has always been a major problem in many pattern recognition and machine learning applications. Trace ratio criterion is a criterion that can be applicable to many dimensionality reduction methods as it directly reflects Euclidean distance between data points of within or between classes. In this paper, we analyze the trace ratio problem and propose a new efficient algorithm to find the optimal solution. Based on the proposed algorithm, we are able to derive an orthogonal constrained semi-supervised learning framework. The new algorithm incorporates unlabeled data into training procedure so that it is able to preserve the discriminative structure as well as geometrical structure embedded in the original dataset. Under such a framework, many existing semi-supervised dimensionality reduction methods such as SDA, Lap-LDA, SSDR, SSMMC, can be improved using our proposed framework, which can also be used to formulate a corresponding kernel framework for handling nonlinear problems. Theoretical analysis indicates that there are certain relationships between linear and nonlinear methods. Finally, extensive simulations on synthetic dataset and real world dataset are presented to show the effectiveness of our algorithms. The results demonstrate that our proposed algorithm can achieve great superiority to other state-of-art algorithms.     

Orthogonal margin discriminant projection for dimensionality reduction

Dimensionality reduction aims to represent high-dimensional data with much smaller number of features, which plays as a preprocessing step to remove the insignificant and irrelevant features in many machine learning applications, resulting in lower computational cost and better performance of classifiers. In most cases, the data points can be well classified with margin samples which are defined as furthest intra-class samples and nearest inter-class samples. Motivated by this observation, this paper proposes a linear supervised dimensionality reduction method called orthogonal margin discriminant projection (OMDP). After OMDP projection, intra-class data points become more compact and inter-class data points become more separated. Extensive experiments have been conducted to evaluate the proposed OMDP algorithm using several benchmark face data sets. The experimental results confirm the effectiveness of the proposed method.     

Trace ratio criterion for multi-view discriminant analysis

Applied Intelligence - Learning from heterogeneous views, termed multi-view learning (MvL), is a significant yet challenging problem in computer vision. Many existing MvL methods apply the two-view...     

Feature extraction using orthogonal discriminant local tangent space alignment

A novel algorithm called orthogonal discriminant local tangent space alignment (O-DLTSA) is proposed for supervised feature extraction. Derived from local tangent space alignment (LTSA), O-DLTSA not only inherits the advantages of LTSA which uses local tangent space as a representation of the local geometry so as to preserve the local structure, but also makes full use of class information and orthogonal subspace to improve discriminant power. The experimental results of applying O-DLTSA to standard face databases demonstrate the effectiveness of the proposed method.     

Local uncorrelated local discriminant embedding for face recognition

The feature extraction algorithm plays an important role in face recognition. However, the extracted features also have overlapping discriminant information. A property of the statistical uncorrelated criterion is that it eliminates the redundancy among the extracted discriminant features, while many algorithms generally ignore this property. In this paper, we introduce a novel feature extraction method called local uncorrelated local discriminant embedding (LULDE). The proposed approach can be seen as an extension of a local discriminant embedding (LDE) framework in three ways. First, a new local statistical uncorrelated criterion is proposed, which effectively captures the local information of interclass and intraclass. Second, we reconstruct the affinity matrices of an intrinsic graph and a penalty graph, which are mentioned in LDE to enhance the discriminant property. Finally, it overcomes the small-sample-size problem without using principal component analysis to preprocess the original data, which avoids losing some discriminant information. Experimental results on Yale, ORL, Extended Yale B, and FERET databases demonstrate that LULDE outperforms LDE and other representative uncorrelated feature extraction methods.     

Random walk-based fuzzy linear discriminant analysis for dimensionality reduction

Dealing with high-dimensional data has always been a major problem with the research of pattern recognition and machine learning, and linear discriminant analysis (LDA) is one of the most popular methods for dimensionality reduction. However, it suffers from the problem of being too sensitive to outliers. Hence to solve this problem, fuzzy membership can be introduced to enhance the performance of algorithms by reducing the effects of outliers. In this paper, we analyze the existing fuzzy strategies and propose a new effective one based on Markov random walks. The new fuzzy strategy can maintain high consistency of local and global discriminative information and preserve statistical properties of dataset. In addition, based on the proposed fuzzy strategy, we then derive an efficient fuzzy LDA algorithm by incorporating the fuzzy membership into learning. Theoretical analysis and extensive simulations show the effectiveness of our algorithm. The presented results demonstrate that our proposed algorithm can achieve significantly improved results compared with other existing algorithms.     

Extracting the optimal dimensionality for local tensor discriminant analysis

Supervised dimensionality reduction with tensor representation has attracted great interest in recent years. It has been successfully applied to problems with tensor data, such as image and video recognition tasks. However, in the tensor-based methods, how to select the suitable dimensions is a very important problem. Since the number of possible dimension combinations exponentially increases with respect to the order of tensor, manually selecting the suitable dimensions becomes an impossible task in the case of high-order tensor. In this paper, we aim at solving this important problem and propose an algorithm to extract the optimal dimensionality for local tensor discriminant analysis. Experimental results on a toy example and real-world data validate the effectiveness of the proposed method.     

Unsupervised double weight graphs based discriminant analysis for dimensionality reduction

ABSTRACT

Dimensionality reduction plays an important role in pattern recognition tasks. Locality preserving projection and neighbourhood preserving embedding are popular unsupervised feature extraction methods, which try to preserve a certain local structure in the low-dimensional subspace. However, only considering the local neighbour information will limit the methods to achieve higher recognition accuracy. In this paper, an unsupervised double weight graphs based discriminant analysis method (uDWG-DA) is proposed. First, uDWG-DA considers both similar and dissimilar relationships among samples by using double weight graphs. In order to explore the dissimilar information, a new partitioning strategy is proposed to divide the data set into different clusters, where samples of different clusters are dissimilar. Then, based on L_2,1 norm, uDWG-DA finds the optimal projection to not only preserve the similar local structure but also increase the separability among different clusters of the data set. Experiments on four hyperspectral images validate the advantage and feasibility of the proposed method compared with other dimensionality reduction methods.     

A new quality assessment criterion for nonlinear dimensionality reduction

A new quality assessment criterion for evaluating the performance of the nonlinear dimensionality reduction (NLDR) methods is proposed in this paper. Differing from the current quality assessment criteria focusing on the local-neighborhood-preserving performance of the NLDR methods, the proposed criterion capitalizes on a new aspect, the global-structure-holding performance, of the NLDR methods. By taking both properties into consideration, the intrinsic capability of the NLDR methods can be more faithfully reflected, and hence more rational measurement for the proper selection of NLDR methods in real-life applications can be offered. The theoretical argument is supported by experiment results implemented on a series of benchmark data sets.     

Graph embedding and extensions: a general framework for dimensionality reduction

A large family of algorithms - supervised or unsupervised; stemming from statistics or geometry theory - has been designed to provide different solutions to the problem of dimensionality reduction. Despite the different motivations of these algorithms, we present in this paper a general formulation known as graph embedding to unify them within a common framework. In graph embedding, each algorithm can be considered as the direct graph embedding or its linear/kernel/tensor extension of a specific intrinsic graph that describes certain desired statistical or geometric properties of a data set, with constraints from scale normalization or a penalty graph that characterizes a statistical or geometric property that should be avoided. Furthermore, the graph embedding framework can be used as a general platform for developing new dimensionality reduction algorithms. By utilizing this framework as a tool, we propose a new supervised dimensionality reduction algorithm called marginal Fisher analysis in which the intrinsic graph characterizes the intraclass compactness and connects each data point with its neighboring points of the same class, while the penalty graph connects the marginal points and characterizes the interclass separability. We show that MFA effectively overcomes the limitations of the traditional linear discriminant analysis algorithm due to data distribution assumptions and available projection directions. Real face recognition experiments show the superiority of our proposed MFA in comparison to LDA, also for corresponding kernel and tensor extensions     

Generalized null space uncorrelated Fisher discriminant analysis for linear dimensionality reduction

We propose a generalized null space uncorrelated Fisher discriminant analysis (GNUFDA) technique integrating the uncorrelated discriminant analysis and weighted pairwise Fisher criterion. The GNUFDA can effectively deal with the small sample-size problem and perform satisfactorily when the dimensionality of the null space decreases with increase in the number of training samples per class and/or classes, C. The proposed GNUFDA can extract at most C-1 optimal uncorrelated discriminative vectors without being influenced by the null-space dimensionality.     

Uncorrelated trace ratio linear discriminant analysis for undersampled problems

For linear discriminant analysis (LDA), the ratio trace and trace ratio are two basic criteria generalized from the classical Fisher criterion function, while the orthogonal and uncorrelated constraints are two common conditions imposed on the optimal linear transformation. The ratio trace criterion with both the orthogonal and uncorrelated constraints have been extensively studied in the literature, whereas the trace ratio criterion receives less interest mainly due to the lack of a closed-form solution and efficient algorithms. In this paper, we make an extensive study on the uncorrelated trace ratio linear discriminant analysis, with particular emphasis on the application on the undersampled problem. Two regularization uncorrelated trace ratio LDA models are discussed for which the global solutions are characterized and efficient algorithms are established. Experimental comparison on several LDA approaches are conducted on several real world datasets, and the results show that the uncorrelated trace ratio LDA is competitive with the orthogonal trace ratio LDA, but is better than the results based on ratio trace criteria in terms of the classification performance.     

Discriminative unsupervised 2D dimensionality reduction with graph embedding

Dimensionality reduction is a great challenge in high dimensional unlabelled data processing. The existing dimensionality reduction methods are prone to employing similarity matrix and spectral clustering algorithm. However, the noises in original data always make the similarity matrix unreliable and degrade the clustering performance. Besides, existing spectral clustering methods just focus on the local structures and ignore the global discriminative information, which may lead to overfitting in some cases. To address these issues, a novel unsupervised 2-dimensional dimensionality reduction method is proposed in this paper, which incorporates the similarity matrix learning and global discriminant information into the procedure of dimensionality reduction. Particularly, the number of the connected components in the learned similarity matrix is equal to cluster number. We compare the proposed method with several 2-dimensional unsupervised dimensionality reduction methods and evaluate the clustering performance by K-means on several benchmark data sets. The experimental results show that the proposed method outperforms the state-of-the-art methods.     

Discover latent discriminant information for dimensionality reduction: Non-negative Sparseness Preserving Embedding

How to define sparse affinity weight matrices is still an open problem in existing manifold learning algorithms. In this paper, we propose a novel unsupervised learning method called Non-negative Sparseness Preserving Embedding (NSPE) for linear dimensionality reduction. Differing from the manifold learning-based subspace learning methods such as Locality Preserving Projections (LPP), Neighbor Preserving Embedding (NPE) and the recently proposed sparse representation based Sparsity Preserving Projections (SPP); NSPE preserves the non-negative sparse reconstruction relationships in low-dimensional subspace. Another novelty of NSPE is the sparseness constraint, which is directly added to control the non-negative sparse representation coefficients. This gives a more ground truth model to imitate the actions of the active neuron cells of V1 of the primate visual cortex on information processing. Although labels are not used in the training steps, the non-negative sparse representation can still discover the latent discriminant information and thus provides better measure coefficients and significant discriminant abilities for feature extraction. Moreover, NSPE is more efficient than the recently proposed sparse representation based SPP algorithm. Comprehensive comparison and extensive experiments show that NSPE has the competitive performance against the unsupervised learning algorithms such as classical PCA and the state-of-the-art techniques: LPP, NPE and SPP.     

Integrating local and global topological structures for semi-supervised dimensionality reduction

Dimensionality reduction plays an important role in many machine learning tasks. This paper studies semi-supervised dimensionality reduction using pairwise constraints. In this setting, domain knowledge is given in the form of pairwise constraint, which specifies whether a pair of instances belongs to the same class (must-link constraint) or different classes (cannot-link constraint). In this paper, a novel semi-supervised dimensionality reduction method called LGS3DR is proposed, which can integrate both local and global topological structures of the data as well as pairwise constraints. The LGS3DR method is effective and has a closed form solution. Experiments on data visualization and face recognition show that LGS3DR is superior to many existing dimensionality reduction methods.     

Geometrically local embedding in manifolds for dimension reduction

In this paper, geometrically local embedding (GLE) is presented to discover the intrinsic structure of manifolds as a method in nonlinear dimension reduction. GLE is able to reveal the inner features of the input data in the lower dimension space while suppressing the influence of outliers in the local linear manifold. In addition to feature extraction and representation, GLE behaves as a clustering and classification method by projecting the feature data into low-dimensional separable regions. Through empirical evaluation, the performance of GLE is demonstrated by the visualization of synthetic data in lower dimension, and the comparison with other dimension reduction algorithms with the same data and configuration. Experiments on both pure and noisy data prove the effectiveness of GLE in dimension reduction, feature extraction, data visualization as well as clustering and classification.     

Local binary pattern-based discriminant graph construction for dimensionality reduction with application to face recognition

Multimedia Tools and Applications - Graph construction has attracted increasing interest in recent years due to its key role in many dimensionality reduction (DR) algorithms. On the other hand, our...     

基于局部Fisher准则判别投影的人脸识别算法

为了提取更真实的样本局部分布结构以及合理利用样本标签信息,提出局部Fisher准则判别投影的人脸识别算法。通过求解样本在总体下稀疏表示来自适应选择样本的近邻参数,以使样本间分布关系尽可能符合真实情况;在获取稀疏近邻结构的基础上,利用样本标签信息设计自定义的类内局部散度矩阵和类间局部散度矩阵,以使得在保留样本间近邻关系的同时提高样本标签信息带来的判别能力。该算法可以有效保持同类样本间的稀疏近邻关系,并且破坏非同类样本间的稀疏近邻结构。在Yale库、AR库以及Yale B库上的实验结果表明:与相关的人脸识别算法相比,该算法具有更高的人脸识别率,可以有效提升人脸识别算法的识别率。     

A dimensionality reduction technique based on a least squared error criterion

A method of achieving dimensionality reduction is presented. The reduced dimensionality is achieved by utilizing a least squared error technique under the assumption that the goodness criterion is the maximum separation of classes. The criterion is met by first maximizing the spread of the cluster centers, and then minimizing the within class scatter. The derivation of the desired transformation from an arbitrary p-space to a space of lower dimension, say l, is completed with the assumption that the cluster centers are known. The criterion for the cluster center location is the minimization of the variance of the distance between the cluster center and the transformed pattern. It is demonstrated that the resulting cluster center set is similar to the simplex signal set in communication theory, which is a minimum energy signal set.