Dual local learning regularized nonnegative matrix factorization and its semi-supervised extension for clustering

Neural Computing and Applications - Nonnegative matrix factorization (NMF) has received considerable attention in data representation due to its strong interpretability. However, traditional NMF...     

Non-negative matrix factorization for semi-supervised data clustering

Traditional clustering algorithms are inapplicable to many real-world problems where limited knowledge from domain experts is available. Incorporating the domain knowledge can guide a clustering algorithm, consequently improving the quality of clustering. In this paper, we propose SS-NMF: a semi-supervised non-negative matrix factorization framework for data clustering. In SS-NMF, users are able to provide supervision for clustering in terms of pairwise constraints on a few data objects specifying whether they “must” or “cannot” be clustered together. Through an iterative algorithm, we perform symmetric tri-factorization of the data similarity matrix to infer the clusters. Theoretically, we show the correctness and convergence of SS-NMF. Moveover, we show that SS-NMF provides a general framework for semi-supervised clustering. Existing approaches can be considered as special cases of it. Through extensive experiments conducted on publicly available datasets, we demonstrate the superior performance of SS-NMF for clustering.

Effective semi-supervised document clustering via active learning with instance-level constraints

Semi-supervised document clustering, which takes into account limited supervised data to group unlabeled documents into clusters, has received significant interest recently. Because of getting supervised data may be expensive, it is important to get most informative knowledge to improve the clustering performance. This paper presents a semi-supervised document clustering algorithm and a new method for actively selecting informative instance-level constraints to get improved clustering performance. The semi- supervised document clustering algorithm is a Constrained DBSCAN (Cons-DBSCAN) algorithm, which incorporates instance-level constraints to guide the clustering process in DBSCAN. An active learning approach is proposed to select informative document pairs for obtaining user feedbacks. Experimental results show that Cons-DBSCAN with our proposed active learning approach can improve the clustering performance significantly when given a relatively small amount of constraints.     

Combined new nonnegative matrix factorization algorithms with two-dimensional nonnegative matrix factorization for image processing

In recent years, nonnegative matrix factorization (NMF) has attracted significant amount of attentions in image processing, text mining, speech processing and related fields. Although NMF has been applied in several application successfully, its simple application on image processing has a few caveats. For example, NMF costs considerable computational resources when performing on large databases. In this paper, we propose two enhanced NMF algorithms for image processing to save the computational costs. One is modified rank-one residue iteration (MRRI) algorithm , the other is element-wisely residue iteration (ERI) algorithm. Here we combine CAPG (a NMF algorithm proposed by Lin), MRRI and ERI with two-dimensional nonnegative matrix factorization (2DNMF) for image processing. The main difference between NMF and 2DNMF is that the former first aligns images into one-dimensional (1D) vectors and then represents them with a set of 1D bases, while the latter regards images as 2D matrices and represents them with a set of 2D bases. The three combined algorithms are named CAPG-2DNMF, MRRI-2DNMF and ERI-2DNMF. The computational complexity and convergence analyses of proposed algorithms are also presented in this paper. Three public databases are used to test the three NMF algorithms and the three combinations, the results of which show the enhancement performance of our proposed algorithms (MRRI and ERI algorithms) over the CAPG algorithm. MRRI and ERI have similar performance. The three combined algorithms have better image reconstruction quality and less running time than their corresponding 1DNMF algorithms under the same compression ratio. We also do some experiments on a real-captured image database and get similar conclusions.     

Symmetric nonnegative matrix factorization with elastic-net regularized block-wise weighted representation for clustering

Pattern Analysis and Applications - In unsupervised learning, symmetric nonnegative matrix factorization (NMF) has proven its efficacy for various clustering tasks in recent years, considering both...     

Correntropy-based dual graph regularized nonnegative matrix factorization with Lp smoothness for data representation

Applied Intelligence - Nonnegative matrix factorization methods have been widely used in many applications in recent years. However, the clustering performances of such methods may deteriorate...     

Quadratic nonnegative matrix factorization

In Nonnegative Matrix Factorization (NMF), a nonnegative matrix is approximated by a product of lower-rank factorizing matrices. Most NMF methods assume that each factorizing matrix appears only once in the approximation, thus the approximation is linear in the factorizing matrices. We present a new class of approximative NMF methods, called Quadratic Nonnegative Matrix Factorization (QNMF), where some factorizing matrices occur twice in the approximation. We demonstrate QNMF solutions to four potential pattern recognition problems in graph partitioning, two-way clustering, estimating hidden Markov chains, and graph matching. We derive multiplicative algorithms that monotonically decrease the approximation error under a variety of measures. We also present extensions in which one of the factorizing matrices is constrained to be orthogonal or stochastic. Empirical studies show that for certain application scenarios, QNMF is more advantageous than other existing nonnegative matrix factorization methods.     

A generalized divergence measure for nonnegative matrix factorization

This letter presents a general parametric divergence measure. The metric includes as special cases quadratic error and Kullback-Leibler divergence. A parametric generalization of the two different multiplicative update rules for nonnegative matrix factorization by Lee and Seung (2001) is shown to lead to locally optimal solutions of the nonnegative matrix factorization problem with this new cost function. Numeric simulations demonstrate that the new update rule may improve the quadratic distance convergence speed. A proof of convergence is given that, as in Lee and Seung, uses an auxiliary function known from the expectation-maximization theoretical framework.     

Manifold-respecting discriminant nonnegative matrix factorization

Nonnegative matrix factorization (NMF) is an unsupervised learning method for low-rank approximation of nonnegative data, where the target matrix is approximated by a product of two nonnegative factor matrices. Two important ingredients are missing in the standard NMF methods: (1) discriminant analysis with label information; (2) geometric structure (manifold) in the data. Most of the existing variants of NMF incorporate one of these ingredients into the factorization. In this paper, we present a variation of NMF which is equipped with both these ingredients, such that the data manifold is respected and label information is incorporated into the NMF. To this end, we regularize NMF by intra-class and inter-class k-nearest neighbor (k-NN) graphs, leading to NMF-kNN, where we minimize the approximation error while contracting intra-class neighborhoods and expanding inter-class neighborhoods in the decomposition. We develop simple multiplicative updates for NMF-kNN and present monotonic convergence results. Experiments on several benchmark face and document datasets confirm the useful behavior of our proposed method in the task of feature extraction.     

Projected gradient methods for nonnegative matrix factorization

Nonnegative matrix factorization (NMF) can be formulated as a minimization problem with bound constraints. Although bound-constrained optimization has been studied extensively in both theory and practice, so far no study has formally applied its techniques to NMF. In this letter, we propose two projected gradient methods for NMF, both of which exhibit strong optimization properties. We discuss efficient implementations and demonstrate that one of the proposed methods converges faster than the popular multiplicative update approach. A simple Matlab code is also provided.     

Using underapproximations for sparse nonnegative matrix factorization

Nonnegative matrix factorization consists in (approximately) factorizing a nonnegative data matrix by the product of two low-rank nonnegative matrices. It has been successfully applied as a data analysis technique in numerous domains, e.g., text mining, image processing, microarray data analysis, collaborative filtering, etc.We introduce a novel approach to solve NMF problems, based on the use of an underapproximation technique, and show its effectiveness to obtain sparse solutions. This approach, based on Lagrangian relaxation, allows the resolution of NMF problems in a recursive fashion. We also prove that the underapproximation problem is NP-hard for any fixed factorization rank, using a reduction of the maximum edge biclique problem in bipartite graphs.We test two variants of our underapproximation approach on several standard image datasets and show that they provide sparse part-based representations with low reconstruction error. Our results are comparable and sometimes superior to those obtained by two standard sparse nonnegative matrix factorization techniques.     

Community discovery using nonnegative matrix factorization

Complex networks exist in a wide range of real world systems, such as social networks, technological networks, and biological networks. During the last decades, many researchers have concentrated on exploring some common things contained in those large networks include the small-world property, power-law degree distributions, and network connectivity. In this paper, we will investigate another important issue, community discovery, in network analysis. We choose Nonnegative Matrix Factorization (NMF) as our tool to find the communities because of its powerful interpretability and close relationship between clustering methods. Targeting different types of networks (undirected, directed and compound), we propose three NMF techniques (Symmetric NMF, Asymmetric NMF and Joint NMF). The correctness and convergence properties of those algorithms are also studied. Finally the experiments on real world networks are presented to show the effectiveness of the proposed methods.     

A sparse nonnegative matrix factorization technique for graph matching problems

Graph matching problem that incorporates pairwise constraints can be cast as an Integer Quadratic Programming (IQP). Since it is NP-hard, approximate methods are required. In this paper, a new approximate method based on nonnegative matrix factorization with sparse constraints is presented. Firstly, the graph matching is formulated as an optimization problem with nonnegative and sparse constraints, followed by an efficient algorithm to solve this constrained problem. Then, we show the strong relationship between the sparsity of the relaxation solution and its effectiveness for graph matching based on our model. A key benefit of our method is that the solution is sparse and thus can approximately impose the one-to-one mapping constraints in the optimization process naturally. Therefore, our method can approximate the original IQP problem more closely than other approximate methods. Extensive and comparative experimental results on both synthetic and real-world data demonstrate the effectiveness of our graph matching method.     

Rank-two residue iteration method for nonnegative matrix factorization

Rank-one residue iteration (RRI) is a recently developed block coordinate method for nonnegative matrix factorization (NMF). Numerical results show that the decomposed matrices generated by RRI method may have several columns, which are zero vectors. In this paper, by studying two special kinds of quadratic programming, we develop two block coordinate methods for NMF, rank-two residue iteration (RTRI) method and rank-two modified residue iteration (RTMRI) method. In the two algorithms, the exact solution of the subproblem can be obtained directly. We also provide that the consequence generated by our proposed algorithms can converge to a stationary point. Numerical results show that the RTRI method and the RTMRI method can yield better solutions, especially RTMRI method can remedy the limitation of the RRI method.     

Algorithms and applications for approximate nonnegative matrix factorization

The development and use of low-rank approximate nonnegative matrix factorization (NMF) algorithms for feature extraction and identification in the fields of text mining and spectral data analysis are presented. The evolution and convergence properties of hybrid methods based on both sparsity and smoothness constraints for the resulting nonnegative matrix factors are discussed. The interpretability of NMF outputs in specific contexts are provided along with opportunities for future work in the modification of NMF algorithms for large-scale and time-varying data sets.     

SVD based initialization: A head start for nonnegative matrix factorization

We describe Nonnegative Double Singular Value Decomposition (NNDSVD), a new method designed to enhance the initialization stage of nonnegative matrix factorization (NMF). NNDSVD can readily be combined with existing NMF algorithms. The basic algorithm contains no randomization and is based on two SVD processes, one approximating the data matrix, the other approximating positive sections of the resulting partial SVD factors utilizing an algebraic property of unit rank matrices. Simple practical variants for NMF with dense factors are described. NNDSVD is also well suited to initialize NMF algorithms with sparse factors. Many numerical examples suggest that NNDSVD leads to rapid reduction of the approximation error of many NMF algorithms.     

非负矩阵分解的复杂网络社团检测方法

为有效地检测复杂网络中的社团结构,优化了评估与发现社团的模块密度函数(即D值).通过模块密度的优化进程,证明了模块函数的最大化与非负矩阵分解目标函数(SNMF)的等价性.基于这种等价性,设计了一种新的基于模块密度函SNMF算法,并且讨论了该算法的复杂性.在一个经典的计算机产生的随机网络中检验了该算法,特别地,当社团结构变模糊时,实验结果表明该算法在发现复杂网络社团上是有效的.     

Nonsmooth nonnegative matrix factorization (nsNMF)

We propose a novel nonnegative matrix factorization model that aims at finding localized, part-based, representations of nonnegative multivariate data items. Unlike the classical nonnegative matrix factorization (NMF) technique, this new model, denoted "nonsmooth nonnegative matrix factorization" (nsNMF), corresponds to the optimization of an unambiguous cost function designed to explicitly represent sparseness, in the form of nonsmoothness, which is controlled by a single parameter. In general, this method produces a set of basis and encoding vectors that are not only capable of representing the original data, but they also extract highly focalized patterns, which generally lend themselves to improved interpretability. The properties of this new method are illustrated with several data sets. Comparisons to previously published methods show that the new nsNMF method has some advantages in keeping faithfulness to the data in the achieving a high degree of sparseness for both the estimated basis and the encoding vectors and in better interpretability of the factors.     

A cooperative semi-supervised fuzzy clustering framework for dental X-ray image segmentation

Dental X-ray image segmentation (DXIS) is an indispensable process in practical dentistry for diagnosis of periodontitis diseases from an X-ray image. It has been said that DXIS is one of the most important and necessary steps to analyze dental images in order to get valuable information for medical diagnosis support systems and other recognition tools. Specialized data mining methods for DXIS have been investigated to achieve high accuracy of segmentation. However, traditional image processing and clustering algorithms often meet challenges in determining parameters or common boundaries of teeth samples. It was shown that performance of a clustering algorithm is enhanced when additional information provided by users is attached to inputs of the algorithm. In this paper, we propose a new cooperative scheme that applies semi-supervised fuzzy clustering algorithms to DXIS. Specifically, the Otsu method is used to remove the Background area from an X-ray dental image. Then, the FCM algorithm is chosen to remove the Dental Structure area from the results of the previous steps. Finally, Semi-supervised Entropy regularized Fuzzy Clustering algorithm (eSFCM) is opted to clarify and improve the results based on the optimal result from the previous clustering method. The proposed framework is evaluated on a real collection of dental X-ray image datasets from Hanoi Medical University, Vietnam. Experimental results have revealed that clustering quality of the cooperative framework is better than those of the relevant ones. The findings of this paper have great impact and significance to researches in the fields of medical science and expert systems. It has been the fact that medical diagnosis is often an experienced and case-based process which requests long time practicing in real patients. In many situations, young clinicians do not have chance for such the practice so that it is necessary to utilize a computerized medical diagnosis system which could simulate medical processes from previous real evidences. By learning from those cases, clinicians would improve their experience and responses for later ones. In the view of expert systems, this paper made uses of knowledge-based algorithms for a practical application. This shows the advantages of such the algorithm in the conjunction domain between expert systems and medical informatics. The findings also suggested the most appropriate configuration of the algorithm and parameters for this problem that could be reused by other researchers in similar applications. The usefulness and significance of this research are clearly demonstrated within the extent of real-life applications.