Subspace learning for unsupervised feature selection via matrix factorization

Dimensionality reduction is an important and challenging task in machine learning and data mining. Feature selection and feature extraction are two commonly used techniques for decreasing dimensionality of the data and increasing efficiency of learning algorithms. Specifically, feature selection realized in the absence of class labels, namely unsupervised feature selection, is challenging and interesting. In this paper, we propose a new unsupervised feature selection criterion developed from the viewpoint of subspace learning, which is treated as a matrix factorization problem. The advantages of this work are four-fold. First, dwelling on the technique of matrix factorization, a unified framework is established for feature selection, feature extraction and clustering. Second, an iterative update algorithm is provided via matrix factorization, which is an efficient technique to deal with high-dimensional data. Third, an effective method for feature selection with numeric data is put forward, instead of drawing support from the discretization process. Fourth, this new criterion provides a sound foundation for embedding kernel tricks into feature selection. With this regard, an algorithm based on kernel methods is also proposed. The algorithms are compared with four state-of-the-art feature selection methods using six publicly available datasets. Experimental results demonstrate that in terms of clustering results, the proposed two algorithms come with better performance than the others for almost all datasets we experimented with here.     

Isolation kernel: the X factor in efficient and effective large scale online kernel learning

Large scale online kernel learning aims to build an efficient and scalable kernel-based predictive model incrementally from a sequence of potentially infinite data points. Current state-of-the-art large scale online kernel learning focuses on improving efficiency. Two key approaches to gain efficiency through approximation are (1) limiting the number of support vectors, and (2) using an approximate feature map. They often employ a kernel with a feature map with intractable dimensionality. While these approaches can deal with large scale datasets efficiently, this outcome is achieved by compromising predictive accuracy because of the approximation. We offer an alternative approach that puts the kernel used at the heart of the approach. It focuses on creating a sparse and finite-dimensional feature map of a kernel called Isolation Kernel. Using this new approach, to achieve the above aim of large scale online kernel learning becomes extremely simple—simply use Isolation Kernel instead of a kernel having a feature map with intractable dimensionality. We show that, using Isolation Kernel, large scale online kernel learning can be achieved efficiently without sacrificing accuracy.

     

Learning bounds for kernel regression using effective data dimensionality

Kernel methods can embed finite-dimensional data into infinite-dimensional feature spaces. In spite of the large underlying feature dimensionality, kernel methods can achieve good generalization ability. This observation is often wrongly interpreted, and it has been used to argue that kernel learning can magically avoid the "curse-of-dimensionality" phenomenon encountered in statistical estimation problems. This letter shows that although using kernel representation, one can embed data into an infinite-dimensional feature space; the effective dimensionality of this embedding, which determines the learning complexity of the underlying kernel machine, is usually small. In particular, we introduce an algebraic definition of a scale-sensitive effective dimension associated with a kernel representation. Based on this quantity, we derive upper bounds on the generalization performance of some kernel regression methods. Moreover, we show that the resulting convergent rates are optimal under various circumstances.     

机器人自学习控制的稳定性分析方法及其应用

提出了一种针对自学习控制的稳定性判据,应用这一稳定性判据将自学习控制器的 设计转化为寻找正定离散矩阵核,从而回答了两个问题,其一什么样的量可以通过自学习叠 代加以控制,其二学习叠代中什么样的滤波环节的引入不会影响学习收敛性.根据这一判据 设计了一种机器人参数自学习控制律,它保证跟踪轨线全程的收敛性.     

桥梁结构的未标记模态特征稀疏编码深度学习监测*

由于建筑物结构健康问题大部分是累积性损害,很难被检测到,实际结构和环境噪声的复杂性使得结构健康监测更加困难,并且现有方法在训练模型时需要大量的数据,但是实际中对于数据的标记是很复杂的。为克服该问题,通过配备无线传感器网络,并采用稀疏编码实现桥梁结构健康监测,然后通过大量未标记实例在实现特征提取基础上进行稀疏编码算法训练,实现数据维度压缩和无标记数据预处理。其次,利用深度学习算法实现桥梁结构健康监测类别预测,同时基于线性共轭梯度对Hessian优化进行改进,利用半正定高斯-牛顿曲率矩阵替换不确定Hessian矩阵,进行二次目标组合,以实现深度学习算法效率提升;实验结果表明,所提深度学习桥梁结构安全检测算法实现了环境噪声稀疏编码水平下的高精度结构健康监测。     

The matrix ridge approximation: algorithms and applications

We are concerned with an approximation problem for a symmetric positive semidefinite matrix due to motivation from a class of nonlinear machine learning methods. We discuss an approximation approach that we call matrix ridge approximation. In particular, we define the matrix ridge approximation as an incomplete matrix factorization plus a ridge term. Moreover, we present probabilistic interpretations using a normal latent variable model and a Wishart model for this approximation approach. The idea behind the latent variable model in turn leads us to an efficient EM iterative method for handling the matrix ridge approximation problem. Finally, we illustrate the applications of the approximation approach in multivariate data analysis. Empirical studies in spectral clustering and Gaussian process regression show that the matrix ridge approximation with the EM iteration is potentially useful.     

A simulation study using EFA and CFA programs based the impact of missing data on test dimensionality

This study examines the impact of missing rates and data imputation methods on test dimensionality. We consider how missing rate levels (10%, 20%, 30%, and 50%) and the six missed data imputation methods (Listwise, Serial Mean, Linear Interpolation, Linear Trend, EM, and Regression) affect the structure of a test. A simulation study is conducted using the SPSS 15.0 EFA and CFA programs. The EFA results for the six methods are similar, and all results obtained two factors. The CFA results also fit the hypothesized two factor structure model for all six methods. However, we observed that the EM method fits the EFA results relatively well. When the percentage of missing data is less than 20%, the impact of the imputation methods on test dimensionality is not statistically significant. The Serial Mean and Linear Trend methods are suggested for use when the percentage of missing data is greater than 30%.     

混杂数据的多核几何平均度量学习

在机器学习和模式识别任务中,选择一种合适的距离度量方法是至关重要的.度量学习主要利用判别性信息学习一个马氏距离或相似性度量.然而,大多数现有的度量学习方法都是针对数值型数据的,对于一些有结构的数据（比如符号型数据）,用传统的距离度量来度量两个对象之间的相似性是不合理的;其次,大多数度量学习方法会受到维度的困扰,高维度使得训练时间长,模型的可扩展性差.提出了一种基于几何平均的混杂数据度量学习方法.采用不同的核函数将数值型数据和符号型数据分别映射到可再生核希尔伯特空间,从而避免了特征的高维度带来的负面影响.同时,提出了一个基于几何平均的多核度量学习模型,将混杂数据的度量学习问题转化为求黎曼流形上两个点的中心点问题.在UCI数据集上的实验结果表明,针对混杂数据的多核度量学习方法与现有的度量学习方法相比,在准确性方面展现出更优异的性能.     

Kernel combination via debiased object correspondence analysis

This paper addresses the problem of combining multi-modal kernels in situations in which object correspondence information is unavailable between modalities, for instance, where missing feature values exist, or when using proprietary databases in multi-modal biometrics. The method thus seeks to recover inter-modality kernel information so as to enable classifiers to be built within a composite embedding space. This is achieved through a principled group-wise identification of objects within differing modal kernel matrices in order to form a composite kernel matrix that retains the full freedom of linear kernel combination existing in multiple kernel learning. The underlying principle is derived from the notion of tomographic reconstruction, which has been applied successfully in conventional pattern recognition.In setting out this method, we aim to improve upon object-correspondence insensitive methods, such as kernel matrix combination via the Cartesian product of object sets to which the method defaults in the case of no discovered pairwise object identifications. We benchmark the method against the augmented kernel method, an order-insensitive approach derived from the direct sum of constituent kernel matrices, and also against straightforward additive kernel combination where the correspondence information is given a priori. We find that the proposed method gives rise to substantial performance improvements.     

On Learning and Cross-Validation with Decomposed Nyström Approximation of Kernel Matrix

The high computational costs of training kernel methods to solve nonlinear tasks limits their applicability. However, recently several fast training methods have been introduced for solving linear learning tasks. These can be used to solve nonlinear tasks by mapping the input data nonlinearly to a low-dimensional feature space. In this work, we consider the mapping induced by decomposing the Nyström approximation of the kernel matrix. We collect together prior results and derive new ones to show how to efficiently train, make predictions with and do cross-validation for reduced set approximations of learning algorithms, given an efficient linear solver. Specifically, we present an efficient method for removing basis vectors from the mapping, which we show to be important when performing cross-validation.     

局部切空间对齐算法的核主成分分析解释

基于核方法的降维技术和流形学习是两类有效而广泛应用的非线性降维技术,它们有着各自不同的出发点和理论基础,在以往的研究中很少有研究关注两者的联系。LTSA算法利用数据的局部结构构造一种特殊的核矩阵,然后利用该核矩阵进行核主成分分析。本文针对局部切空间对齐这种流形学习算法,重点研究了LTSA算法与核PCA的内在联系。研究表明,LTSA在本质上是一种基于核方法的主成分分析技术。     

基于双重局部保持的不完整多视角嵌入学习方法

现有的多视角降维方法多数假设数据是完整的,但该假设在实际应用中难以实现。为解决不完整多视角数据降维问题,提出一种新的不完整多视角嵌入学习方法。基于多视角数据的一致性与同一视角下样本间的线性相关性学习一组重构系数,对缺失样本进行线性重构,通过学习所有视角的公共低维嵌入,保持原始空间的局部几何结构。在此基础上,设计一种惩罚参数来度量重构样本的可靠度,从而权衡缺失样本对学习结果的负面影响。实验结果表明,该方法在Yale、ORL和COIL-20数据集上NMI值分别达到65.63%、73.23%和78.27%,较MVL-IV算法分别提升8.37%、16.71%和20.24%。     

Multi-layer radial basis function neural network based on multi-scale kernel learning

The conversion functions in the hidden layer of radial basis function neural networks (RBFNN) are Gaussian functions. The Gaussian functions are local to the kernel centers. In most of the existing research, the spatial local response of the sample is inaccurately calculated because the kernels have the same shape as a hypersphere, and the kernel parameters in the network are determined by experience. The influence of the fine structure in the local space is not considered during feature extraction. In addition, it is difficult to obtain a better feature extraction ability with less computational complexity. Therefore, this paper develops a multi-scale RBF kernel learning algorithm and proposes a new multi-layer RBF neural network model. For the samples of each class, the expectation maximization (EM) algorithm is used to obtain multi-layer nested sub-distribution models with different local response ranges, which are called multi-scale kernels in the network. The prior information of each sub-distribution is used as the connection weight between the multi-scale kernels. Finally, feature extraction is implemented using multi-layer kernel subspace embedding. The multi-scale kernel learning model can efficiently and accurately describe the fine structure of the samples and is fault tolerant to setting the number of kernels to a certain extent. Considering the prior probability of each kernel as the weight makes the feature extraction process satisfy the Bayes rule, which can enhance the interpretability of feature extraction in the network. This paper also theoretically proves that the proposed neural network is a generalized version of the original RBFNN. The experimental results show that the proposed method has better performance compared with some state-of-the-art algorithms.     

Kernel Spectral Matched Filter for Hyperspectral Imagery

In this paper a kernel-based nonlinear spectral matched filter is introduced for target detection in hyperspectral imagery, which is implemented by using the ideas in kernel-based learning theory. A spectral matched filter is defined in a feature space of high dimensionality, which is implicitly generated by a nonlinear mapping associated with a kernel function. A kernel version of the matched filter is derived by expressing the spectral matched filter in terms of the vector dot products form and replacing each dot product with a kernel function using the so called kernel trick property of the Mercer kernels. The proposed kernel spectral matched filter is equivalent to a nonlinear matched filter in the original input space, which is capable of generating nonlinear decision boundaries. The kernel version of the linear spectral matched filter is implemented and simulation results on hyperspectral imagery show that the kernel spectral matched filter outperforms the conventional linear matched filter.     

Accelerating EM for Large Databases

The EM algorithm is a popular method for parameter estimation in a variety of problems involving missing data. However, the EM algorithm often requires significant computational resources and has been dismissed as impractical for large databases. We present two approaches that significantly reduce the computational cost of applying the EM algorithm to databases with a large number of cases, including databases with large dimensionality. Both approaches are based on partial E-steps for which we can use the results of Neal and Hinton (In Jordan, M. (Ed.), Learning in Graphical Models, pp. 355–371. The Netherlands: Kluwer Academic Publishers) to obtain the standard convergence guarantees of EM. The first approach is a version of the incremental EM algorithm, described in Neal and Hinton (1998), which cycles through data cases in blocks. The number of cases in each block dramatically effects the efficiency of the algorithm. We provide amethod for selecting a near optimal block size. The second approach, which we call lazy EM, will, at scheduled iterations, evaluate the significance of each data case and then proceed for several iterations actively using only the significant cases. We demonstrate that both methods can significantly reduce computational costs through their application to high-dimensional real-world and synthetic mixture modeling problems for large databases.     

核参数判别选择方法在核主元分析中的应用

针对核主元分析(KPCA)中高斯核参数β的经验选取问题,提出了核主元分析的核参数判别选择方法。依据训练样本的类标签计算类内、类间核窗宽,在以上核窗宽中经判别选择方法确定核参数。根据判别选择核参数所确定的核矩阵,能够准确描述训练空间的结构特征。用主成分分析(PCA)对特征空间进行分解,提取主成分以实现降维和特征提取。判别核窗宽方法在分类密集区域选择较小窗宽,在分类稀疏区域选择较大窗宽。将判别核主成分分析(Dis-KPCA)应用到数据模拟实例和田纳西过程(TEP),通过与KPCA、PCA方法比较,实验结果表明,Dis-KPCA方法有效地对样本数据降维且将三个类别数据100%分开,因此,所提方法的降维精度更高。     

Implementation of machine-learning classification in remote sensing: an applied review

Machine learning offers the potential for effective and efficient classification of remotely sensed imagery. The strengths of machine learning include the capacity to handle data of high dimensionality and to map classes with very complex characteristics. Nevertheless, implementing a machine-learning classification is not straightforward, and the literature provides conflicting advice regarding many key issues. This article therefore provides an overview of machine learning from an applied perspective. We focus on the relatively mature methods of support vector machines, single decision trees (DTs), Random Forests, boosted DTs, artificial neural networks, and k-nearest neighbours (k-NN). Issues considered include the choice of algorithm, training data requirements, user-defined parameter selection and optimization, feature space impacts and reduction, and computational costs. We illustrate these issues through applying machine-learning classification to two publically available remotely sensed data sets.     

Missing multi-label learning with non-equilibrium based on two-level autoencoder

For a multi-label learning framework, each instance may belong to multiple labels simultaneously. The classification accuracy can be improved significantly by exploiting various correlations, such as label correlations, feature correlations, or the correlations between features and labels. There are few studies on how to combine the feature and label correlations, and they deal more with complete data sets. However, missing labels or other phenomena often occur because of the cost or technical limitations in the data acquisition process. A few label completion algorithms currently suitable for missing multi-label learning, ignore the noise interference of the feature space. At the same time, the threshold of the discriminant function often affects the classification results, especially those of the labels near the threshold. All these factors pose considerable difficulties in dealing with missing labels using label correlations. Therefore, we propose a missing multi-label learning algorithm with non-equilibrium based on a two-level autoencoder. First, label density is introduced to enlarge the classification margin of the label space. Then, a new supplementary label matrix is augmented from the missing label matrix with the non-equilibrium label completion method. Finally, considering feature space noise, a two-level kernel extreme learning machine autoencoder is constructed to implement the information feature and label correlation. The effectiveness of the proposed algorithm is verified by many experiments on both missing and complete label data sets. A statistical analysis of hypothesis validates our approach.

     

Intelligent Intrusion Detection System for Industrial Internet of Things Environment

Rapid increase in the large quantity of industrial data, Industry 4.0/5.0 poses several challenging issues such as heterogeneous data generation, data sensing and collection, real-time data processing, and high request arrival rates. The classical intrusion detection system (IDS) is not a practical solution to the Industry 4.0 environment owing to the resource limitations and complexity. To resolve these issues, this paper designs a new Chaotic Cuckoo Search Optimization Algorithm (CCSOA) with optimal wavelet kernel extreme learning machine (OWKELM) named CCSOA-OWKELM technique for IDS on the Industry 4.0 platform. The CCSOA-OWKELM technique focuses on the design of feature selection with classification approach to achieve minimum computation complexity and maximum detection accuracy. The CCSOA-OWKELM technique involves the design of CCSOA based feature selection technique, which incorporates the concepts of chaotic maps with CSOA. Besides, the OWKELM technique is applied for the intrusion detection and classification process. In addition, the OWKELM technique is derived by the hyperparameter tuning of the WKELM technique by the use of sunflower optimization (SFO) algorithm. The utilization of CCSOA for feature subset selection and SFO algorithm based hyperparameter tuning leads to better performance. In order to guarantee the supreme performance of the CCSOA-OWKELM technique, a wide range of experiments take place on two benchmark datasets and the experimental outcomes demonstrate the promising performance of the CCSOA-OWKELM technique over the recent state of art techniques.     

基于KLLDA和ELM的新型模拟电路故障诊断方法

模拟电路是工业设备中最重要的元器件,其故障可能造成重大的人员伤亡,甚至造成巨大的经济损失。针对这一问题,提出一种基于核局部线性判别分析(Kernel Local Linear Discriminant Analysis,KLLDA)的故障诊断方案。利用小波分析和统计分析对原始信号进行预处理,得到原始特征集;利用KLLDA方法进行降维,并与核主成分分析(Kernel Principal Component Analysis,KPCA)和核线性判别分析(Kernel Linear Discriminant Analysis,KLDA)方法进行比较;采用极限学习机(Extreme Learning Machine,ELM)对测试电路的故障进行定位。对两个故障诊断案例的实验结果表明了该方法的有效性,并表明KLLDA在降维方面总体上优于KPCA和KLDA。