Dm-KDE: dynamical kernel density estimation by sequences of KDE estimators with fixed number of components over data streams

In many data stream mining applications, traditional density estimation methods such as kernel density estimation, reduced set density estimation can not be applied to the density estimation of data streams because of their high computational burden, processing time and intensive memory allocation requirement. In order to reduce the time and space complexity, a novel density estimation method Dm-KDE over data streams based on the proposed algorithm m-KDE which can be used to design a KDE estimator with the fixed number of kernel components for a dataset is proposed. In this method, Dm-KDE sequence entries are created by algorithm m-KDE instead of all kernels obtained from other density estimation methods. In order to further reduce the storage space, Dm-KDE sequence entries can be merged by calculating their KL divergences. Finally, the probability density functions over arbitrary time or entire time can be estimated through the obtained estimation model. In contrast to the state-of-the-art algorithm SOMKE, the distinctive advantage of the proposed algorithm Dm-KDE exists in that it can achieve the same accuracy with much less fixed number of kernel components such that it is suitable for the scenarios where higher on-line computation about the kernel density estimation over data streams is required.We compare Dm-KDE with SOMKE and M-kernel in terms of density estimation accuracy and running time for various stationary datasets. We also apply Dm-KDE to evolving data streams. Experimental results illustrate the effectiveness of the proposed method.     

参数可调的通用半正交图像矩模型

目的 为了提高以正交多项式为核函数构造的高阶矩数值的稳定性,增强低阶矩抗噪和滤波的能力,将仅具有全局描述能力的常规正交矩推广到可以局部化提取图像特征的矩模型,从频率特性分析的角度定义一种参数可调的通用半正交矩模型。方法 首先,对传统正交矩的核函数进行合理的修正,以修正后的核函数（也称基函数）替代传统正交矩中的原核函数,使其成为修改后的特例之一。经过修正后的基函数可以有效消除图像矩数值不稳定现象。其次,采用时域的分析方法能够对图像的低阶矩作定量的分析,但无法对图像的高频部分（对应的高阶矩）作更合理的表述。因此提出一种时—频对应的方法来分析和增强不同阶矩的稳定性,通过对修正后核函数的频带宽度微调可以建立性能更优的不同阶矩。最后,利用构建的半正交—三角函数矩研究和分析了通用半正交矩模型的特点及性质。结果 将三角函数为核函数的图像矩与现有的Zernike、伪Zernike、正交傅里叶—梅林矩及贝塞尔—傅里叶矩相比,由于核函数组成简单,且其值域恒定在[-1,1]区间,因此在图像识别领域具有更快的计算速度和更高的稳定性。结论 理论分析和一系列相关图像的仿真实验表明,与传统的正交矩相比,在数值稳定性、图像重构、图像感兴趣区域（ROI）特征检测、噪声鲁棒性测试及不变性识别方面,通用的半正交矩性能及效果更优。     

New recursive estimators from correlated interrupted observations using covariance information

Least-squares linear and quadratic filtering and fixed-point smoothing algorithms are derived to estimate a signal from uncertain observations perturbed by an additive white noise. The random variables describing the uncertainty are correlated only at consecutive time instants, and this correlation, as well as the probability that the signal exists in each observation, is known. Recursive algorithms are obtained without requiring the state-space model generating the signal, but just some moments of both the signal and the additive noise in the observation equation. For the linear estimation algorithms, only the second-order moments are required, and the autocovariance function of the signal must be expressed in a semi-degenerate kernel form. The quadratic estimation algorithms use, in addition, the moments up to the fourth one, and they require the autocovariance and cross-covariance functions of the signal and their second-order powers in a semi-degenerate kernel form. This form for expressing autocovariance functions is not very restrictive, since it covers many general stochastic processes, including stationary and non-stationary processes.     

Median radial basis function neural network

Radial basis functions (RBFs) consist of a two-layer neural network, where each hidden unit implements a kernel function. Each kernel is associated with an activation region from the input space and its output is fed to an output unit. In order to find the parameters of a neural network which embeds this structure we take into consideration two different statistical approaches. The first approach uses classical estimation in the learning stage and it is based on the learning vector quantization algorithm and its second-order statistics extension. After the presentation of this approach, we introduce the median radial basis function (MRBF) algorithm based on robust estimation of the hidden unit parameters. The proposed algorithm employs the marginal median for kernel location estimation and the median of the absolute deviations for the scale parameter estimation. A histogram-based fast implementation is provided for the MRBF algorithm. The theoretical performance of the two training algorithms is comparatively evaluated when estimating the network weights. The network is applied in pattern classification problems and in optical flow segmentation.     

Hybrid wavelet ν-support vector machine and chaotic particle swarm optimization for regression estimation

In view of the bad approximate results of the existing support vector (SV) kernel for series influenced by multi-factors in quadratic continuous integral space, combining wavelet theory with kernel technique, a wavelet kernel function is put forward in quadratic continuous integral space. And then, wavelet ν-support vector machine (W ν-SVM) with wavelet kernel is proposed. To seek the optimal parameters of W ν-SVM, embedded chaotic particle swarm optimization (ECPSO) is also proposed to optimize parameters of W ν-SVM. The results of application in car sale estimation show that the estimation approach based on the W ν-SVM and ECPSO is effective and feasible. Compared with the traditional model, W ν-SVM method requires fewer samples and has better estimating precision.     

Evolutionary strategies for hyperparameters of support vector machines based on multi-scale radial basis function kernels

Kernel functions are used in support vector machines (SVM) to compute inner product in a higher dimensional feature space. SVM classification performance depends on the chosen kernel. The radial basis function (RBF) kernel is a distance-based kernel that has been successfully applied in many tasks. This paper focuses on improving the accuracy of SVM by proposing a non-linear combination of multiple RBF kernels to obtain more flexible kernel functions. Multi-scale RBF kernels are weighted and combined. The proposed kernel allows better discrimination in the feature space. This new kernel is proved to be a Mercer’s kernel. Furthermore, evolutionary strategies (ESs) are used for adjusting the hyperparameters of SVM. Training accuracy, the bound of generalization error, and subset cross-validation on training accuracy are considered to be objective functions in the evolutionary process. The experimental results show that the accuracy of multi-scale RBF kernels is better than that of a single RBF kernel. Moreover, the subset cross-validation on training accuracy is more suitable and it yields the good results on benchmark datasets.     

基于高效核偏最小二乘的质量相关故障检测

核偏最小二乘(KPLS)是一种多元统计方法, 广泛应用于过程监控, 然而, KPLS采用斜交分解, 导致质量相关空间存在冗余信息易引发误报警. 因此, 本文提出了高效核偏最小二乘(EKPLS)模型, 所提方法通过奇异值分解(SVD)将核矩阵正交分解为质量相关空间和质量无关空间, 有效降低质量相关空间中的冗余信息, 并采用主成分分析(PCA)按方差大小将质量相关空间分解为质量主空间和质量次空间. 此外, 为进一步降低由质量无关故障引发的误报警, 提出基于质量估计的正交信号修正(OSC)预处理方法, 并结合EKPLS模型提出了OSC-EKPLS算法. OSCEKPLS通过质量估计值对被测数据进行OSC预处理, 降低了计算复杂度和误报率. 最后, 通过数值仿真和田纳西–伊斯曼过程验证了OSC-EKPLS具有良好的故障检测性和更低的误报率.     

Gaussian kernels for incomplete data

This paper discusses a method to estimate the expected value of the Gaussian kernel in the presence of incomplete data. We show how, under the general assumption of a missing-at-random mechanism, the expected value of the Gaussian kernel function has a simple closed-form solution. Such a solution depends only on the parameters of the Gamma distribution which is assumed to represent squared distances. Furthermore, we show how the parameters governing the Gamma distribution depend only on the non-central moments of the kernel arguments, via the second-order moments of their squared distance, and can be estimated by making use of any parametric density estimation model of the data distribution. We approximate the data distribution with the maximum likelihood estimate of a Gaussian mixture distribution. The validity of the method is empirically assessed, under a range of conditions, on synthetic and real problems and the results compared to existing methods. For comparison, we consider methods that indirectly estimate a Gaussian kernel function by either estimating squared distances or by imputing missing values and then computing distances. Based on the experimental results, the proposed method consistently proves itself an accurate technique that further extends the use of Gaussian kernels with incomplete data.     

Preservation of local linearity by neighborhood subspace scaling for solving the pre-image problem

An important issue involved in kernel methods is the pre-image problem. However, it is an ill-posed problem, as the solution is usually nonexistent or not unique. In contrast to direct methods aimed at minimizing the distance in feature space, indirect methods aimed at constructing approximate equivalent models have shown outstanding performance. In this paper, an indirect method for solving the pre-image problem is proposed. In the proposed algorithm, an inverse mapping process is constructed based on a novel framework that preserves local linearity. In this framework, a local nonlinear transformation is implicitly conducted by neighborhood subspace scaling transformation to preserve the local linearity between feature space and input space. By extending the inverse mapping process to test samples, we can obtain pre-images in input space. The proposed method is non-iterative, and can be used for any kernel functions. Experimental results based on image denoising using kernel principal component analysis （PCA） show that the proposed method outperforms the state-of-the-art methods for solving the pre-image problem.     

Choosing parameters of kernel subspace LDA for recognition of face images under pose and illumination variations.

This paper addresses the problem of automatically tuning multiple kernel parameters for the kernel-based linear discriminant analysis (LDA) method. The kernel approach has been proposed to solve face recognition problems under complex distribution by mapping the input space to a high-dimensional feature space. Some recognition algorithms such as the kernel principal components analysis, kernel Fisher discriminant, generalized discriminant analysis, and kernel direct LDA have been developed in the last five years. The experimental results show that the kernel-based method is a good and feasible approach to tackle the pose and illumination variations. One of the crucial factors in the kernel approach is the selection of kernel parameters, which highly affects the generalization capability and stability of the kernel-based learning methods. In view of this, we propose an eigenvalue-stability-bounded margin maximization (ESBMM) algorithm to automatically tune the multiple parameters of the Gaussian radial basis function kernel for the kernel subspace LDA (KSLDA) method, which is developed based on our previously developed subspace LDA method. The ESBMM algorithm improves the generalization capability of the kernel-based LDA method by maximizing the margin maximization criterion while maintaining the eigenvalue stability of the kernel-based LDA method. An in-depth investigation on the generalization performance on pose and illumination dimensions is performed using the YaleB and CMU PIE databases. The FERET database is also used for benchmark evaluation. Compared with the existing PCA-based and LDA-based methods, our proposed KSLDA method, with the ESBMM kernel parameter estimation algorithm, gives superior performance.     

基于加权核独立成分分析的故障检测方法

针对核独立成分分析故障检测时忽略各独立成分分量对系统故障贡献度的差异, 提出一种基于加权核独立成分分析的故障检测方法. 使用核独立成分分析提取过程变量的独立成分, 根据核密度估计衡量各独立成分分量对系统故障的贡献度, 对各独立成分分量赋予不同权重, 突出包含有用信息的独立成分分量, 引入局部离群因子在特征空间构造统计量进行故障检测. 基于数值仿真和Tennessee Eastman 数据集的仿真结果表明了所提出方法的优越性.

     

基于特征空间k最近邻的批次过程监视

针对具有非高斯、非线性及多工况特性的批次过程,提出一种基于特征量最近邻统计指标的过程监视方法. 首先,将批次过程正常工况原始数据投影到其特征空间,提取主元T和平方预测误差SPE,并进行特征量k最近邻距离平方和的求解. 然后,采用核密度估计法获得概率密度分布函数,确定统计监视控制限. 特征空间的主元T和SPE特征量能全面代表原始数据的有用信息. 采用特征量k最近邻建立监视模型将会节省存储空间,提高建模样本数量与变量之比以及检测异常工况的速度. 另外,利用局部近邻数据建模可以解决过程具有的非线性和多工况问题,而应用核密度估计法可以解决过程数据具有的非高斯分布问题. 最后,在半导体生产过程的成功应用表明了所提方法的有效性.     

Edit distance-based kernel functions for structural pattern classification

A common approach in structural pattern classification is to define a dissimilarity measure on patterns and apply a distance-based nearest-neighbor classifier. In this paper, we introduce an alternative method for classification using kernel functions based on edit distance. The proposed approach is applicable to both string and graph representations of patterns. By means of the kernel functions introduced in this paper, string and graph classification can be performed in an implicit vector space using powerful statistical algorithms. The validity of the kernel method cannot be established for edit distance in general. However, by evaluating theoretical criteria we show that the kernel functions are nevertheless suitable for classification, and experiments on various string and graph datasets clearly demonstrate that nearest-neighbor classifiers can be outperformed by support vector machines using the proposed kernel functions.     

一种快速计算Zernike矩的改进q-递归算法

提出了一种快速计算Zernike矩的改进q-递归算法,该方法通过同时降低核函数中Zernike多项式和Fourier函数的计算复杂度以提高Zernike矩的计算效率。采用 q-递归法快速计算Zernike多项式以避免复杂的阶乘运算,再利用x轴、y轴、x=y和x=-y 4条直线将图像域分成8等分。计算Zernike矩时,仅计算其中1个区域的核函数的值,其他区域的值可以通过核函数关于4条直线的对称性得到。该方法不仅减少了核函数的存储空间,而且大大降低了Zernike矩的计算时间。试验结果表明,与现有方法相比,改进q-递归算法具有更好的性能。     

A new approach to fuzzy modeling of nonlinear dynamic systems with noise: relevance vector learning mechanism

This paper presents a new fuzzy inference system for modeling of nonlinear dynamic systems based on input and output data with measurement noise. The proposed fuzzy system has a number of fuzzy rules and parameter values of membership functions which are automatically generated using the extended relevance vector machine (RVM). The RVM has a probabilistic Bayesian learning framework and has good generalization capability. The RVM consists of the sum of product of weight and kernel function which projects input space into high dimensional feature space. The structure of proposed fuzzy system is same as that of the Takagi-Sugeno fuzzy model. However, in the proposed method, the number of fuzzy rules can be reduced under the process of optimizing a marginal likelihood by adjusting parameter values of kernel functions using the gradient ascent method. After a fuzzy system is determined, coefficients in consequent part are found by the least square method. Examples illustrate effectiveness of the proposed new fuzzy inference system.     

Nonlinear process monitoring using kernel dictionary learning with application to aluminum electrolysis process

In practice, because of complex mechanism processes, such as heating process, volume heterogeneity, and various chemical reaction characteristics, there is a nonlinear relationship among variables in industrial systems. The nonlinearity brings some difficulties to process monitoring. In order to ensure that the process monitoring system can work normally in nonlinear production processes, the nonlinear relationship between variables ought to be considered. In this work, a new fault detection and isolation method based on kernel dictionary learning is presented. In detail, the linearly inseparable data is mapped to a high-dimensional space. Then, a new nonlinear dictionary learning method based on kernel method was proposed to learn the dictionary. After obtaining the dictionary, the control limit can be calculated from the training data according to the kernel density estimation (KDE) method. When new data arrive, they can be represented by the well-learned dictionary, and the kernel reconstruction error can be used as a classifier for process monitoring. As for the fault data, the iterative reconstruction based method is proposed for fault isolation. In order to evaluate the effectiveness of the proposed process monitoring method, some extensive experiments on a numerical simulation, the continuous stirred tank heater (CSTH) process, and a real industrial aluminum electrolysis process are conducted. The proposed method is compared with several state-of-the-art process monitoring methods and the experimental results show that the proposed method can provide satisfactory monitoring results, especially for some small faults, thus it is suitable for process monitoring of nonlinear industrial processes.     

Multi-layer radial basis function neural network based on multi-scale kernel learning

The conversion functions in the hidden layer of radial basis function neural networks (RBFNN) are Gaussian functions. The Gaussian functions are local to the kernel centers. In most of the existing research, the spatial local response of the sample is inaccurately calculated because the kernels have the same shape as a hypersphere, and the kernel parameters in the network are determined by experience. The influence of the fine structure in the local space is not considered during feature extraction. In addition, it is difficult to obtain a better feature extraction ability with less computational complexity. Therefore, this paper develops a multi-scale RBF kernel learning algorithm and proposes a new multi-layer RBF neural network model. For the samples of each class, the expectation maximization (EM) algorithm is used to obtain multi-layer nested sub-distribution models with different local response ranges, which are called multi-scale kernels in the network. The prior information of each sub-distribution is used as the connection weight between the multi-scale kernels. Finally, feature extraction is implemented using multi-layer kernel subspace embedding. The multi-scale kernel learning model can efficiently and accurately describe the fine structure of the samples and is fault tolerant to setting the number of kernels to a certain extent. Considering the prior probability of each kernel as the weight makes the feature extraction process satisfy the Bayes rule, which can enhance the interpretability of feature extraction in the network. This paper also theoretically proves that the proposed neural network is a generalized version of the original RBFNN. The experimental results show that the proposed method has better performance compared with some state-of-the-art algorithms.     

Domain-Specific Modeling for Rapid Energy Estimation of Reconfigurable Architectures

Reconfigurable architectures such as FPGAs are flexible alternatives to DSPs or ASICs used in mobile devices for which energy is a key performance metric. Reconfigurable architectures offer several design parameters such as operating frequency, precision, amount of memory, degree of parallelism, etc. These parameters define a large design space that must be explored to find energy-efficient solutions. It is also challenging to predict the energy variation at the early design phases when a design is modified at algorithm level. Efficient traversal of such a large design space requires high-level modeling to facilitate rapid estimation of system-wide energy. However, FPGAs do not exhibit a high-level structure like, for example, a RISC processor for which high-level as well as low-level energy models are available. To address this scenario, we propose a domain-specific modeling technique for energy-efficient kernel design that exploits the knowledge of the algorithm and the target architecture family for a given kernel to develop a high-level model. This model captures architecture and algorithm features, parameters affecting energy performance, and power estimation functions based on these parameters. A system-wide energy function is derived based on the power functions and cycle specific power state of each building block of the architecture. This model is used to understand the impact of various parameters on system-wide energy and can be a basis for the design of energy-efficient algorithms. Our high-level model is used to quickly obtain fairly accurate estimate of the system-wide energy dissipation of data paths configured using FPGAs. We demonstrate our modeling methodology by applying it to four domains.     

Gaussian kernel-based fuzzy inference systems for high dimensional regression

We propose a novel architecture for a higher order fuzzy inference system (FIS) and develop a learning algorithm to build the FIS. The consequent part of the proposed FIS is expressed as a nonlinear combination of the input variables, which can be obtained by introducing an implicit mapping from the input space to a high dimensional feature space. The proposed learning algorithm consists of two phases. In the first phase, the antecedent fuzzy sets are estimated by the kernel-based fuzzy c-means clustering. In the second phase, the consequent parameters are identified by support vector machine whose kernel function is constructed by fuzzy membership functions and the Gaussian kernel. The performance of the proposed model is verified through several numerical examples generally used in fuzzy modeling. Comparative analysis shows that, compared with the zero-order fuzzy model, first-order fuzzy model, and polynomial fuzzy model, the proposed model exhibits higher accuracy, better generalization performance, and satisfactory robustness.     

Image Analysis by Two Types of Franklin-Fourier Moments

In this paper, we first derive two types of transformed Franklin polynomial: substituted and weighted radial Franklin polynomials. Two radial orthogonal moments are proposed based on these two types of polynomials, namely substituted Franklin-Fourier moments and weighted Franklin-Fourier moments (SFFMs and WFFMs), which are orthogonal in polar coordinates. The radial kernel functions of SFFMs and WFFMs are transformed Franklin functions and Franklin functions are composed of a class of complete orthogonal splines function system of degree one. Therefore, it provides the possibility of avoiding calculating high order polynomials, and thus the accurate values of SFFMs and WFFMs can be obtained directly with little computational cost. Theoretical and experimental results show that Franklin functions are not well suited for constructing higher-order moments of SFFMs and WFFMs, but compared with traditional orthogonal moments (e.g., BFMs, OFMs and ZMs) in polar coordinates, the proposed two types of Franklin-Fourier Moments have better performance respectively in lower-order moments.