Initialization enhancer for non-negative matrix factorization

Non-negative matrix factorization (NMF), proposed recently by Lee and Seung, has been applied to many areas such as dimensionality reduction, image classification image compression, and so on. Based on traditional NMF, researchers have put forward several new algorithms to improve its performance. However, particular emphasis has to be placed on the initialization of NMF because of its local convergence, although it is usually ignored in many documents. In this paper, we explore three initialization methods based on principal component analysis (PCA), fuzzy clustering and Gabor wavelets either for the consideration of computational complexity or the preservation of structure. In addition, the three methods develop an efficient way of selecting the rank of the NMF in low-dimensional space.     

Source separation using regularized NMF with MMSE estimates under GMM priors with online learning for the uncertainties

We propose a new method to incorporate priors on the solution of nonnegative matrix factorization (NMF). The NMF solution is guided to follow the minimum mean square error (MMSE) estimates of the weight combinations under a Gaussian mixture model (GMM) prior. The proposed algorithm can be used for denoising or single-channel source separation (SCSS) applications. NMF is used in SCSS in two main stages, the training stage and the separation stage. In the training stage, NMF is used to decompose the training data spectrogram for each source into a multiplication of a trained basis and gains matrices. In the separation stage, the mixed signal spectrogram is decomposed as a weighted linear combination of the trained basis matrices for the source signals. In this work, to improve the separation performance of NMF, the trained gains matrices are used to guide the solution of the NMF weights during the separation stage. The trained gains matrix is used to train a prior GMM that captures the statistics of the valid weight combinations that the columns of the basis matrix can receive for a given source signal. In the separation stage, the prior GMMs are used to guide the NMF solution of the gains/weights matrices using MMSE estimation. The NMF decomposition weights matrix is treated as a distorted image by a distortion operator, which is learned directly from the observed signals. The MMSE estimate of the weights matrix under the trained GMM prior and log-normal distribution for the distortion is then found to improve the NMF decomposition results. The MMSE estimate is embedded within the optimization objective to form a novel regularized NMF cost function. The corresponding update rules for the new objectives are derived in this paper. The proposed MMSE estimates based regularization avoids the problem of computing the hyper-parameters and the regularization parameters. MMSE also provides a better estimate for the valid gains matrix. Experimental results show that the proposed regularized NMF algorithm improves the source separation performance compared with using NMF without a prior or with other prior models.     

Robust embedded projective nonnegative matrix factorization for image analysis and feature extraction

Nonnegative matrix factorization (NMF) is an unsupervised learning method for decomposing high-dimensional nonnegative data matrices and extracting basic and intrinsic features. Since image data are described and stored as nonnegative matrices, the mining and analysis process usually involves the use of various NMF strategies. NMF methods have well-known applications in face recognition, image reconstruction, handwritten digit recognition, image denoising and feature extraction. Recently, several projective NMF (P-NMF) methods based on positively constrained projections have been proposed and were found to perform better than the standard NMF approach in some aspects. However, some drawbacks still affect the existing NMF and P-NMF algorithms; these include dense factors, slow convergence, learning poor local features, and low reconstruction accuracy. The aim of this paper is to design algorithms that address the aforementioned issues. In particular, we propose two embedded P-NMF algorithms: the first method combines the alternating least squares (ALS) algorithm with the P-NMF update rules of the Frobenius norm and the second one embeds ALS with the P-NMF update rule of the Kullback–Leibler divergence. To assess the performances of the proposed methods, we conducted various experiments on four well-known data sets of faces. The experimental results reveal that the proposed algorithms outperform other related methods by providing very sparse factors and extracting better localized features. In addition, the empirical studies show that the new methods provide highly orthogonal factors that possess small entropy values.     

SVD based initialization: A head start for nonnegative matrix factorization

We describe Nonnegative Double Singular Value Decomposition (NNDSVD), a new method designed to enhance the initialization stage of nonnegative matrix factorization (NMF). NNDSVD can readily be combined with existing NMF algorithms. The basic algorithm contains no randomization and is based on two SVD processes, one approximating the data matrix, the other approximating positive sections of the resulting partial SVD factors utilizing an algebraic property of unit rank matrices. Simple practical variants for NMF with dense factors are described. NNDSVD is also well suited to initialize NMF algorithms with sparse factors. Many numerical examples suggest that NNDSVD leads to rapid reduction of the approximation error of many NMF algorithms.     

Regularized nonnegative matrix factorization using Gaussian mixture priors for supervised single channel source separation

We introduce a new regularized nonnegative matrix factorization (NMF) method for supervised single-channel source separation (SCSS). We propose a new multi-objective cost function which includes the conventional divergence term for the NMF together with a prior likelihood term. The first term measures the divergence between the observed data and the multiplication of basis and gains matrices. The novel second term encourages the log-normalized gain vectors of the NMF solution to increase their likelihood under a prior Gaussian mixture model (GMM) which is used to encourage the gains to follow certain patterns. In this model, the parameters to be estimated are the basis vectors, the gain vectors and the parameters of the GMM prior. We introduce two different ways to train the model parameters, sequential training and joint training. In sequential training, after finding the basis and gains matrices, the gains matrix is then used to train the prior GMM in a separate step. In joint training, within each NMF iteration the basis matrix, the gains matrix and the prior GMM parameters are updated jointly using the proposed regularized NMF. The normalization of the gains makes the prior models energy independent, which is an advantage as compared to earlier proposals. In addition, GMM is a much richer prior than the previously considered alternatives such as conjugate priors which may not represent the distribution of the gains in the best possible way. In the separation stage after observing the mixed signal, we use the proposed regularized cost function with a combined basis and the GMM priors for all sources that were learned from training data for each source. Only the gain vectors are estimated from the mixed data by minimizing the joint cost function. We introduce novel update rules that solve the optimization problem efficiently for the new regularized NMF problem. This optimization is challenging due to using energy normalization and GMM for prior modeling, which makes the problem highly nonlinear and non-convex. The experimental results show that the introduced methods improve the performance of single channel source separation for speech separation and speech–music separation with different NMF divergence functions. The experimental results also show that, using the GMM prior gives better separation results than using the conjugate prior.     

基于改进型遗传算法的虚拟人上肢运动链逆运动学求解方法

由于人体上肢运动链的高自由度,用传统的几何法、解析法、迭代法等求其逆解较为困难。遗传算法具有很好的寻优特性,但标准遗传算法在求解时容易陷入早熟收敛和后期搜索迟钝。为此,提出了一种改进型遗传算法(IGA)求解的方法。先构建人体上肢运动链的各关节单元,并用D-H方法建立其数学模型;然后仿人类种群现象实现遗传算法的种群多样化和种群初始化,设计具有自适应性能的交叉概率和变异概率算子,从而完成了对标准遗传算法的改进。通过对比仿真计算结果可得,改进后的遗传算法能以更大概率避免陷入早熟收敛和后期搜索迟钝,并以较少的遗传代数寻得高精度逆解。     

鲁棒自适应对称非负矩阵分解聚类算法

对称非负矩阵分解SNMF作为一种基于图的聚类算法,能够更自然地捕获图表示中嵌入的聚类结构,并且在线性和非线性流形上获得更好的聚类结果,但对变量的初始化比较敏感。另外,标准的SNMF算法利用误差平方和来衡量分解的质量,对噪声和异常值敏感。为了解决这些问题,在集成学习视角下,提出一种鲁棒自适应对称非负矩阵分解聚类算法RS³NMF(robust self-adaptived symmetric nonnegative matrix factorization)。基于L_2,1范数的RS³NMF模型缓解了噪声和异常值的影响,保持了特征旋转不变性,提高了模型的鲁棒性。同时,在不借助任何附加信息的前提下,利用SNMF对初始化特征的敏感性来逐步增强聚类性能。采用交替迭代方法优化,并保证目标函数值的收敛性。大量实验结果表明,所提RS³NMF算法优于其他先进的算法,具有较强的鲁棒性。     

基于搜索空间划分与Canopy K-means聚类的种群初始化方法

为了提高差分进化算法对搜索空间的探索与开发能力,提高差分进化算法的收敛性与算法的进化效率,提出一种基于搜索空间均匀划分与局部搜索和聚类相结合的种群初始化方法.该方法首先对决策变量空间进行均匀划分,并从各个子空间中随机选择一个个体,得到的个体能够覆盖整个搜索空间;然后,利用Hooke-Jeeves算法对各子空间进行局部搜索得到局部最优的个体,并结合改进的Canopy算法与K-means聚类算法,辨识搜索空间中的前景区域,以此为基础对局部搜索产生的局部最优个体进行筛选,最终生成初始种群中的个体.通过与其他种群初始化方法对CEC2017中5个测试函数进行实验对比,所提出的方法的运行时间可缩减为已有方法的0.75倍,适应度函数可减少为已有方法的0.03倍,且具有最小的标准差以及最优的收敛特性.     

The Design of a Coprime‐Factorized Predictive Functional Controller for Unstable Fractional Order Systems

In this paper, a new approach, called coprime‐factorized predictive functional control method (CFPFC‐F) is proposed to control unstable fractional order linear time invariant systems. To design the controller, first, a prediction model should be synthesized. For this purpose, coprime‐factorized representation is extended for unstable fractional order systems via a reduced approximated model of unstable fractional order (FO) system. That is, an approximated integer model of fractional order system is derived via the well‐known Oustaloup method. Then, the high order approximated model is reduced to a lower one via a balanced truncation model order reduction method. Next, the equivalent coprime‐factorized model of the unstable fractional‐order plant is employed to predict the output of the system. Then, a predictive functional controller (PFC) is designed to control the unstable plant. Finally, the robust stability of the closed‐loop system is analyzed via small gain theorem. The performance of the proposed control is investigated via simulations for the control of an unstable non‐laminated electromagnetic suspension system as our simulation test system.     

Improving non-negative matrix factorizations through structured initialization

In this paper we explore a recent iterative compression technique called non-negative matrix factorization (NMF). Several special properties are obtained as a result of the constrained optimization problem of NMF. For facial images, the additive nature of NMF results in a basis of features, such as eyes, noses, and lips. We explore various methods for efficiently computing NMF, placing particular emphasis on the initialization of current algorithms. We propose using Spherical K-Means clustering to produce a structured initialization for NMF. We demonstrate some of the properties that result from this initialization and develop an efficient way of choosing the rank of the low-dimensional NMF representation.     

Subclass discriminant Nonnegative Matrix Factorization for facial image analysis

Nonnegative Matrix Factorization (NMF) is among the most popular subspace methods, widely used in a variety of image processing problems. Recently, a discriminant NMF method that incorporates Linear Discriminant Analysis inspired criteria has been proposed, which achieves an efficient decomposition of the provided data to its discriminant parts, thus enhancing classification performance. However, this approach possesses certain limitations, since it assumes that the underlying data distribution is unimodal, which is often unrealistic. To remedy this limitation, we regard that data inside each class have a multimodal distribution, thus forming clusters and use criteria inspired by Clustering based Discriminant Analysis. The proposed method incorporates appropriate discriminant constraints in the NMF decomposition cost function in order to address the problem of finding discriminant projections that enhance class separability in the reduced dimensional projection space, while taking into account subclass information. The developed algorithm has been applied for both facial expression and face recognition on three popular databases. Experimental results verified that it successfully identified discriminant facial parts, thus enhancing recognition performance.     

基于S域NMF的模拟电路故障特征提取新方法

针对提取的模拟电路故障特征向量信息不够充分的问题,提出了一种将S时频变换(ST,S-transform)和非负矩阵分解(NMF,non-negative matrix factorization)相结合的特征选取新方法.该方法先对模拟电路故障响应信号应用S时频变换建立时频图谱矩阵,再用NMF算法构造时频图谱数据集合的子空间基矩阵,有效降低了投影特征向量的维数,保留了足够多的故障隐含特征信息,进而提高模拟电路故障识别率.最后,在Sallen-Key高通滤波器电路中验证了文中方法的有效性.     

Manifold-respecting discriminant nonnegative matrix factorization

Nonnegative matrix factorization (NMF) is an unsupervised learning method for low-rank approximation of nonnegative data, where the target matrix is approximated by a product of two nonnegative factor matrices. Two important ingredients are missing in the standard NMF methods: (1) discriminant analysis with label information; (2) geometric structure (manifold) in the data. Most of the existing variants of NMF incorporate one of these ingredients into the factorization. In this paper, we present a variation of NMF which is equipped with both these ingredients, such that the data manifold is respected and label information is incorporated into the NMF. To this end, we regularize NMF by intra-class and inter-class k-nearest neighbor (k-NN) graphs, leading to NMF-kNN, where we minimize the approximation error while contracting intra-class neighborhoods and expanding inter-class neighborhoods in the decomposition. We develop simple multiplicative updates for NMF-kNN and present monotonic convergence results. Experiments on several benchmark face and document datasets confirm the useful behavior of our proposed method in the task of feature extraction.     

Digital watermarking using Hall property image decomposition method

Most of the existing singular value decomposition-based digital watermarking methods are not robust to geometric rotation, which change the pixels’ locations without maintaining the corresponding changes to the pixel’s intensity values of entire image and yield high computational cost. To answer this, we propose a digital image watermarking algorithm using the Hall property. In the proposed method, a digital watermark image is factorized into lower-triangular, upper-triangular, and permutation matrices. The permutation matrix is used as the valid key matrix for authentication of the rightful ownership of the watermark image. The product of the lower and upper triangular matrices is processed with a few iterations of the Arnold transformation to obtain the scrambled data. The scrambled data are embedded into particular sub-bands of a cover image using Wavelet transform. Our experiments show that the proposed algorithm is highly reliable and computationally efficient compared with state-of-the-art methods that are based on singular value decomposition.     

Snapshot ensembles of non-negative matrix factorization for stability of topic modeling

Recently many topic models such as Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) have made important progress towards generating high-level knowledge from a large corpus. However, these algorithms based on random initialization generate different results on the same corpus using the same parameters, denoted as instability problem. For solving this problem, ensembles of NMF are known to be much more stable and accurate than individual NMFs. However, training multiple NMFs for ensembling is computationally expensive. In this paper, we propose a novel scheme to obtain the seemingly contradictory goal of ensembling multiple NMFs without any additional training cost. We train a single NMF algorithm with the cyclical learning rate schedule, which can converge to several local minima along its optimization path. We save the results to the ensemble when the model converges, and then restart the optimization with a large learning rate that can help escape the current local minimum. Based on experiments performed on text corpora using a number of measures to assess, our method can reduce instability at no additional training cost, while simultaneously yields more accurate topic models than traditional single methods and ensemble methods.     

An iterative genetic algorithm for the assembly line worker assignment and balancing problem of type-II

In this study, we consider the assembly line worker assignment and balancing problem of type-II (ALWABP-2). ALWABP-2 arises when task times differ depending on operator skills and concerns with the assignment of tasks and operators to stations in order to minimize the cycle time. We developed an iterative genetic algorithm (IGA) to solve this problem. In the IGA, three search approaches are adopted in order to obtain search diversity and efficiency: modified bisection search, genetic algorithm and iterated local search. When designing the IGA, all the parameters such as construction heuristics, genetic operators and local search operators are adapted specifically to the ALWABP-2. The performance of the proposed IGA is compared with heuristic and metaheuristic approaches on benchmark problem instances. Experimental results show that the proposed IGA is very effective and robust for a large set of benchmark problems.     

Robust watermarking based on DWT and nonnegative matrix factorization

This paper presents a novel blind robust digital image watermarking scheme using nonnegative matrix factorization (NMF) in DWT domain. Firstly, the original image is transformed into some subband coefficients using discrete wavelet transformation (DWT), and then a Gaussian pseudo-random watermark sequence is embedded in the factorized decomposition coefficients using NMF. Because of the multiresolution decomposition for DWT and physically meaningful factorization for NMF, the proposed scheme can achieve good robustness, which is also demonstrated in the following experiments.     

Receding Horizon Controls for Input-Delayed Systems

This paper presents a receding horizon control (RHC) for an unconstrained input-delayed system. To begin with, we derive a finite horizon optimal control for a quadratic cost function including two final weighting terms. The RHC is easily obtained by changing the initial and final times of the finite horizon optimal control. A linear matrix inequality (LMI) condition on two final weighting matrices is proposed to meet the cost monotonicity, under which the optimal cost on the horizon is monotonically nonincreasing with time and hence the asymptotical stability is guaranteed only if an observability condition is met. It is shown through simulation that the proposed RHC stabilizes the input-delayed system effectively and its performance can be tuned by adjusting weighting matrices with respect to the state and the input.      

一种采用改进CPSO算法的PID参数整定方法

为提高控制系统的性能,提出了一种采用改进混沌粒子群(CPSO)算法的PID参数整定方法。该算法将混沌搜索应用到粒子群算法的粒子位置和速度初始化、惯性权重优化、随机常数以及局部最优解邻域点的产生的全过程,使其不仅具有全局寻优能力,而且具有持续与精细的局部搜索能力。3种典型控制系统的PID参数整定实验结果验证了所提方法的有效性,其性能明显优于常规方法。