Quadratic nonnegative matrix factorization

In Nonnegative Matrix Factorization (NMF), a nonnegative matrix is approximated by a product of lower-rank factorizing matrices. Most NMF methods assume that each factorizing matrix appears only once in the approximation, thus the approximation is linear in the factorizing matrices. We present a new class of approximative NMF methods, called Quadratic Nonnegative Matrix Factorization (QNMF), where some factorizing matrices occur twice in the approximation. We demonstrate QNMF solutions to four potential pattern recognition problems in graph partitioning, two-way clustering, estimating hidden Markov chains, and graph matching. We derive multiplicative algorithms that monotonically decrease the approximation error under a variety of measures. We also present extensions in which one of the factorizing matrices is constrained to be orthogonal or stochastic. Empirical studies show that for certain application scenarios, QNMF is more advantageous than other existing nonnegative matrix factorization methods.     

Localized user-driven topic discovery via boosted ensemble of nonnegative matrix factorization

Nonnegative matrix factorization (NMF) has been widely used in topic modeling of large-scale document corpora, where a set of underlying topics are extracted by a low-rank factor matrix from NMF. However, the resulting topics often convey only general, thus redundant information about the documents rather than information that might be minor, but potentially meaningful to users. To address this problem, we present a novel ensemble method based on nonnegative matrix factorization that discovers meaningful local topics. Our method leverages the idea of an ensemble model, which has shown advantages in supervised learning, into an unsupervised topic modeling context. That is, our model successively performs NMF given a residual matrix obtained from previous stages and generates a sequence of topic sets. The algorithm we employ to update is novel in two aspects. The first lies in utilizing the residual matrix inspired by a state-of-the-art gradient boosting model, and the second stems from applying a sophisticated local weighting scheme on the given matrix to enhance the locality of topics, which in turn delivers high-quality, focused topics of interest to users. We subsequently extend this ensemble model by adding keyword- and document-based user interaction to introduce user-driven topic discovery.     

Combined new nonnegative matrix factorization algorithms with two-dimensional nonnegative matrix factorization for image processing

In recent years, nonnegative matrix factorization (NMF) has attracted significant amount of attentions in image processing, text mining, speech processing and related fields. Although NMF has been applied in several application successfully, its simple application on image processing has a few caveats. For example, NMF costs considerable computational resources when performing on large databases. In this paper, we propose two enhanced NMF algorithms for image processing to save the computational costs. One is modified rank-one residue iteration (MRRI) algorithm , the other is element-wisely residue iteration (ERI) algorithm. Here we combine CAPG (a NMF algorithm proposed by Lin), MRRI and ERI with two-dimensional nonnegative matrix factorization (2DNMF) for image processing. The main difference between NMF and 2DNMF is that the former first aligns images into one-dimensional (1D) vectors and then represents them with a set of 1D bases, while the latter regards images as 2D matrices and represents them with a set of 2D bases. The three combined algorithms are named CAPG-2DNMF, MRRI-2DNMF and ERI-2DNMF. The computational complexity and convergence analyses of proposed algorithms are also presented in this paper. Three public databases are used to test the three NMF algorithms and the three combinations, the results of which show the enhancement performance of our proposed algorithms (MRRI and ERI algorithms) over the CAPG algorithm. MRRI and ERI have similar performance. The three combined algorithms have better image reconstruction quality and less running time than their corresponding 1DNMF algorithms under the same compression ratio. We also do some experiments on a real-captured image database and get similar conclusions.     

Manifold-respecting discriminant nonnegative matrix factorization

Nonnegative matrix factorization (NMF) is an unsupervised learning method for low-rank approximation of nonnegative data, where the target matrix is approximated by a product of two nonnegative factor matrices. Two important ingredients are missing in the standard NMF methods: (1) discriminant analysis with label information; (2) geometric structure (manifold) in the data. Most of the existing variants of NMF incorporate one of these ingredients into the factorization. In this paper, we present a variation of NMF which is equipped with both these ingredients, such that the data manifold is respected and label information is incorporated into the NMF. To this end, we regularize NMF by intra-class and inter-class k-nearest neighbor (k-NN) graphs, leading to NMF-kNN, where we minimize the approximation error while contracting intra-class neighborhoods and expanding inter-class neighborhoods in the decomposition. We develop simple multiplicative updates for NMF-kNN and present monotonic convergence results. Experiments on several benchmark face and document datasets confirm the useful behavior of our proposed method in the task of feature extraction.     

Nonsmooth nonnegative matrix factorization (nsNMF)

We propose a novel nonnegative matrix factorization model that aims at finding localized, part-based, representations of nonnegative multivariate data items. Unlike the classical nonnegative matrix factorization (NMF) technique, this new model, denoted "nonsmooth nonnegative matrix factorization" (nsNMF), corresponds to the optimization of an unambiguous cost function designed to explicitly represent sparseness, in the form of nonsmoothness, which is controlled by a single parameter. In general, this method produces a set of basis and encoding vectors that are not only capable of representing the original data, but they also extract highly focalized patterns, which generally lend themselves to improved interpretability. The properties of this new method are illustrated with several data sets. Comparisons to previously published methods show that the new nsNMF method has some advantages in keeping faithfulness to the data in the achieving a high degree of sparseness for both the estimated basis and the encoding vectors and in better interpretability of the factors.     

Projected gradient methods for nonnegative matrix factorization

Nonnegative matrix factorization (NMF) can be formulated as a minimization problem with bound constraints. Although bound-constrained optimization has been studied extensively in both theory and practice, so far no study has formally applied its techniques to NMF. In this letter, we propose two projected gradient methods for NMF, both of which exhibit strong optimization properties. We discuss efficient implementations and demonstrate that one of the proposed methods converges faster than the popular multiplicative update approach. A simple Matlab code is also provided.     

Using underapproximations for sparse nonnegative matrix factorization

Nonnegative matrix factorization consists in (approximately) factorizing a nonnegative data matrix by the product of two low-rank nonnegative matrices. It has been successfully applied as a data analysis technique in numerous domains, e.g., text mining, image processing, microarray data analysis, collaborative filtering, etc.We introduce a novel approach to solve NMF problems, based on the use of an underapproximation technique, and show its effectiveness to obtain sparse solutions. This approach, based on Lagrangian relaxation, allows the resolution of NMF problems in a recursive fashion. We also prove that the underapproximation problem is NP-hard for any fixed factorization rank, using a reduction of the maximum edge biclique problem in bipartite graphs.We test two variants of our underapproximation approach on several standard image datasets and show that they provide sparse part-based representations with low reconstruction error. Our results are comparable and sometimes superior to those obtained by two standard sparse nonnegative matrix factorization techniques.     

A generalized divergence measure for nonnegative matrix factorization

This letter presents a general parametric divergence measure. The metric includes as special cases quadratic error and Kullback-Leibler divergence. A parametric generalization of the two different multiplicative update rules for nonnegative matrix factorization by Lee and Seung (2001) is shown to lead to locally optimal solutions of the nonnegative matrix factorization problem with this new cost function. Numeric simulations demonstrate that the new update rule may improve the quadratic distance convergence speed. A proof of convergence is given that, as in Lee and Seung, uses an auxiliary function known from the expectation-maximization theoretical framework.     

采用非负矩阵分解的语音盲分离

针对多通路语音信号的欠定卷积混合模型,提出一种基于非负矩阵分解(NMF)的语音盲分离方法。该方法使用高斯分量对源信号的短时傅里叶变换(STFT)进行表示,高斯分量由基于板仓-斋藤(Itakura-Saito(IS))散度的非负矩阵分解的因子所组成。使用极大期望值算法(EM)求解参数,并对信号进行重组。该方法被应用到双声道立体声信号的盲分离实验,实验结果表明了该方法的有效性。     

Algorithms and applications for approximate nonnegative matrix factorization

The development and use of low-rank approximate nonnegative matrix factorization (NMF) algorithms for feature extraction and identification in the fields of text mining and spectral data analysis are presented. The evolution and convergence properties of hybrid methods based on both sparsity and smoothness constraints for the resulting nonnegative matrix factors are discussed. The interpretability of NMF outputs in specific contexts are provided along with opportunities for future work in the modification of NMF algorithms for large-scale and time-varying data sets.     

Rank-two residue iteration method for nonnegative matrix factorization

Rank-one residue iteration (RRI) is a recently developed block coordinate method for nonnegative matrix factorization (NMF). Numerical results show that the decomposed matrices generated by RRI method may have several columns, which are zero vectors. In this paper, by studying two special kinds of quadratic programming, we develop two block coordinate methods for NMF, rank-two residue iteration (RTRI) method and rank-two modified residue iteration (RTMRI) method. In the two algorithms, the exact solution of the subproblem can be obtained directly. We also provide that the consequence generated by our proposed algorithms can converge to a stationary point. Numerical results show that the RTRI method and the RTMRI method can yield better solutions, especially RTMRI method can remedy the limitation of the RRI method.     

非负矩阵分解的复杂网络社团检测方法

为有效地检测复杂网络中的社团结构,优化了评估与发现社团的模块密度函数(即D值).通过模块密度的优化进程,证明了模块函数的最大化与非负矩阵分解目标函数(SNMF)的等价性.基于这种等价性,设计了一种新的基于模块密度函SNMF算法,并且讨论了该算法的复杂性.在一个经典的计算机产生的随机网络中检验了该算法,特别地,当社团结构变模糊时,实验结果表明该算法在发现复杂网络社团上是有效的.     

Nonlinear nonnegative matrix factorization based on Mercer kernel construction

Generalizations ofnonnegative matrix factorization (NMF) in kernel feature space, such as projected gradient kernel NMF (PGKNMF) and polynomial Kernel NMF (PNMF), have been developed for face and facial expression recognition recently. However, these existing kernel NMF approaches cannot guarantee the nonnegativity of bases in kernel feature space and thus are essentially semi-NMF methods. In this paper, we show that nonlinear semi-NMF cannot extract the localized components which offer important information in object recognition. Therefore, nonlinear NMF rather than semi-NMF is needed to be developed for extracting localized component as well as learning the nonlinear structure. In order to address the nonlinear problem of NMF and the semi-nonnegative problem of the existing kernel NMF methods, we develop the nonlinear NMF based on a self-constructed Mercer kernel which preserves the nonnegative constraints on both bases and coefficients in kernel feature space. Experimental results in face and expressing recognition show that the proposed approach outperforms the existing state-of-the-art kernel methods, such as KPCA, GDA, PNMF and PGKNMF.     

Kernel nonnegative matrix factorization for spectral EEG feature extraction

Nonnegative matrix factorization (NMF) seeks a decomposition of a nonnegative matrix X0 into a product of two nonnegative factor matrices U0 and V0, such that a discrepancy between X and UV is minimized. Assuming U=XW in the decomposition (for W0), kernel NMF (KNMF) is easily derived in the framework of least squares optimization. In this paper we make use of KNMF to extract discriminative spectral features from the time–frequency representation of electroencephalogram (EEG) data, which is an important task in EEG classification. Especially when KNMF with linear kernel is used, spectral features are easily computed by a matrix multiplication, while in the standard NMF multiplicative update should be performed repeatedly with the other factor matrix fixed, or the pseudo-inverse of a matrix is required. Moreover in KNMF with linear kernel, one can easily perform feature selection or data selection, because of its sparsity nature. Experiments on two EEG datasets in brain computer interface (BCI) competition indicate the useful behavior of our proposed methods.     

Regularized nonnegative matrix factorization using Gaussian mixture priors for supervised single channel source separation

We introduce a new regularized nonnegative matrix factorization (NMF) method for supervised single-channel source separation (SCSS). We propose a new multi-objective cost function which includes the conventional divergence term for the NMF together with a prior likelihood term. The first term measures the divergence between the observed data and the multiplication of basis and gains matrices. The novel second term encourages the log-normalized gain vectors of the NMF solution to increase their likelihood under a prior Gaussian mixture model (GMM) which is used to encourage the gains to follow certain patterns. In this model, the parameters to be estimated are the basis vectors, the gain vectors and the parameters of the GMM prior. We introduce two different ways to train the model parameters, sequential training and joint training. In sequential training, after finding the basis and gains matrices, the gains matrix is then used to train the prior GMM in a separate step. In joint training, within each NMF iteration the basis matrix, the gains matrix and the prior GMM parameters are updated jointly using the proposed regularized NMF. The normalization of the gains makes the prior models energy independent, which is an advantage as compared to earlier proposals. In addition, GMM is a much richer prior than the previously considered alternatives such as conjugate priors which may not represent the distribution of the gains in the best possible way. In the separation stage after observing the mixed signal, we use the proposed regularized cost function with a combined basis and the GMM priors for all sources that were learned from training data for each source. Only the gain vectors are estimated from the mixed data by minimizing the joint cost function. We introduce novel update rules that solve the optimization problem efficiently for the new regularized NMF problem. This optimization is challenging due to using energy normalization and GMM for prior modeling, which makes the problem highly nonlinear and non-convex. The experimental results show that the introduced methods improve the performance of single channel source separation for speech separation and speech–music separation with different NMF divergence functions. The experimental results also show that, using the GMM prior gives better separation results than using the conjugate prior.     

A sparse nonnegative matrix factorization technique for graph matching problems

Graph matching problem that incorporates pairwise constraints can be cast as an Integer Quadratic Programming (IQP). Since it is NP-hard, approximate methods are required. In this paper, a new approximate method based on nonnegative matrix factorization with sparse constraints is presented. Firstly, the graph matching is formulated as an optimization problem with nonnegative and sparse constraints, followed by an efficient algorithm to solve this constrained problem. Then, we show the strong relationship between the sparsity of the relaxation solution and its effectiveness for graph matching based on our model. A key benefit of our method is that the solution is sparse and thus can approximately impose the one-to-one mapping constraints in the optimization process naturally. Therefore, our method can approximate the original IQP problem more closely than other approximate methods. Extensive and comparative experimental results on both synthetic and real-world data demonstrate the effectiveness of our graph matching method.     

Semi-supervised nonnegative matrix factorization with positive and negative label propagations

Applied Intelligence - Semi-supervised nonnegative matrix factorization (SNMF) methods yield the enhanced representation ability over nonnegative matrix factorization (NMF) by incorporating the...     

Image representation using Laplacian regularized nonnegative tensor factorization

Tensor provides a better representation for image space by avoiding information loss in vectorization. Nonnegative tensor factorization (NTF), whose objective is to express an n-way tensor as a sum of k rank-1 tensors under nonnegative constraints, has recently attracted a lot of attentions for its efficient and meaningful representation. However, NTF only sees Euclidean structures in data space and is not optimized for image representation as image space is believed to be a sub-manifold embedded in high-dimensional ambient space. To avoid the limitation of NTF, we propose a novel Laplacian regularized nonnegative tensor factorization (LRNTF) method for image representation and clustering in this paper. In LRNTF, the image space is represented as a 3-way tensor and we explicitly consider the manifold structure of the image space in factorization. That is, two data points that are close to each other in the intrinsic geometry of image space shall also be close to each other under the factorized basis. To evaluate the performance of LRNTF in image representation and clustering, we compare our algorithm with NMF, NTF, NCut and GNMF methods on three standard image databases. Experimental results demonstrate that LRNTF achieves better image clustering performance, while being more insensitive to noise.