A fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks

In this paper, we present a fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks (FAOS-PFNN), where a novel structure learning algorithm incorporating a pruning strategy into new growth criteria is developed. The proposed growing procedure without pruning not only speeds up the online learning process but also facilitates a more parsimonious fuzzy neural network while achieving comparable performance and accuracy by virtue of the growing and pruning strategy. The FAOS-PFNN starts with no hidden neurons and parsimoniously generates new hidden units according to the proposed growth criteria as learning proceeds. In the parameter learning phase, all the free parameters of hidden units, regardless of whether they are newly created or originally existing, are updated by the extended Kalman filter (EKF) method. The effectiveness and superiority of the FAOS-PFNN paradigm is compared with other popular approaches like resource allocation network (RAN), RAN via the extended Kalman filter (RANEKF), minimal resource allocation network (MRAN), adaptive-network-based fuzzy inference system (ANFIS), orthogonal least squares (OLS), RBF-AFS, dynamic fuzzy neural networks (DFNN), generalized DFNN (GDFNN), generalized GAP-RBF (GGAP-RBF), online sequential extreme learning machine (OS-ELM) and self-organizing fuzzy neural network (SOFNN) on various benchmark problems in the areas of function approximation, nonlinear dynamic system identification, chaotic time-series prediction and real-world regression problems. Simulation results demonstrate that the proposed FAOS-PFNN algorithm can achieve faster learning speed and more compact network structure with comparably high accuracy of approximation and generalization.     

Tracking vocal tract resonances using a quantized nonlinear function embeddedin a temporal constraint

This paper presents a new technique for high-accuracy tracking of vocal-tract resonances (which coincide with formants for nonnasalized vowels) in natural speech. The technique is based on a discretized nonlinear prediction function, which is embedded in a temporal constraint on the quantized input values over adjacent time frames as the prior knowledge for their temporal behavior. The nonlinear prediction is constructed, based on its analytical form derived in detail in this paper, as a parameter-free, discrete mapping function that approximates the "forward" relationship from the resonance frequencies and bandwidths to the Linear Predictive Coding (LPC) cepstra of real speech. Discretization of the function permits the "inversion" of the function via a search operation. We further introduce the nonlinear-prediction residual, characterized by a multivariate Gaussian vector with trainable mean vectors and covariance matrices, to account for the errors due to the functional approximation. We develop and describe an expectation-maximization (EM)-based algorithm for training the parameters of the residual, and a dynamic programming-based algorithm for resonance tracking. Details of the algorithm implementation for computation speedup are provided. Experimental results are presented which demonstrate the effectiveness of our new paradigm for tracking vocal-tract resonances. In particular, we show the effectiveness of training the prediction-residual parameters in obtaining high-accuracy resonance estimates, especially during consonantal closure.     

An adaptive and hybrid artificial bee colony algorithm (aABC) for ANFIS training

In this study, we propose an Adaptive and Hybrid Artificial Bee Colony (aABC) algorithm to train ANFIS. Unlike the standard ABC algorithm, two new parameters are utilized in the solution search equation. These are arithmetic crossover rate and adaptivity coefficient. aABC algorithm gains the rapid convergence feature with the usage of arithmetic crossover and it is applied on two different problem groups and its performance is measured. Firstly, it is performed over 10 numerical ‘benchmark functions’. The results show that aABC algorithm is more efficient than standard ABC algorithm. Secondly, ANFIS is trained by using aABC algorithm to identify the nonlinear dynamic systems. Each application begins with the randomly selected initial population and then average RMSE is obtained. For four examples considered in ANFIS training, train error values are respectively computed as 0.0344, 0.0232, 0.0152 and 0.0205. Also, test error values for these examples are respectively found as 0.0255, 0.0202, 0.0146 and 0.0295. Although it varies according to the examples, performance increase between 4.51% and 33.33% occurs. Additionally, it is seen that aABC algorithm converges bettter than ABC algorithm in the all examples. The obtained results are compared with the neuro-fuzzy based approaches which are commonly used in the literature and it is seen that the proposed ABC variant can be efficiently used for ANFIS training.     

The Bayesian evidence scheme for regularizing probability-density estimating neural networks

Training probability-density estimating neural networks with the expectation-maximization (EM) algorithm aims to maximize the likelihood of the training set and therefore leads to overfitting for sparse data. In this article, a regularization method for mixture models with generalized linear kernel centers is proposed, which adopts the Bayesian evidence approach and optimizes the hyperparameters of the prior by type II maximum likelihood. This includes a marginalization over the parameters, which is done by Laplace approximation and requires the derivation of the Hessian of the log-likelihood function. The incorporation of this approach into the standard training scheme leads to a modified form of the EM algorithm, which includes a regularization term and adapts the hyperparameters on-line after each EM cycle. The article presents applications of this scheme to classification problems, the prediction of stochastic time series, and latent space models.     

On-line EM algorithm for the normalized gaussian network

A normalized gaussian network (NGnet) (Moody & Darken, 1989) is a network of local linear regression units. The model softly partitions the input space by normalized gaussian functions, and each local unit linearly approximates the output within the partition. In this article, we propose a new on-line EMalgorithm for the NGnet, which is derived from the batch EMalgorithm (Xu, Jordan, &Hinton 1995), by introducing a discount factor. We show that the on-line EM algorithm is equivalent to the batch EM algorithm if a specific scheduling of the discount factor is employed. In addition, we show that the on-line EM algorithm can be considered as a stochastic approximation method to find the maximum likelihood estimator. A new regularization method is proposed in order to deal with a singular input distribution. In order to manage dynamic environments, where the input-output distribution of data changes over time, unit manipulation mechanisms such as unit production, unit deletion, and unit division are also introduced based on probabilistic interpretation. Experimental results show that our approach is suitable for function approximation problems in dynamic environments. We also apply our on-line EM algorithm to robot dynamics problems and compare our algorithm with the mixtures-of-experts family.     

混沌时间序列的自适应变异差分进化ANFIS预测

提出了一种基于自适应变异差分进化（AMDE）算法的ANFIS模型对混沌时间序列进行预测的方法,该方法采用自适应变异差分进化算法和最小二乘法相结合的混合学习算法对ANFIS网络结构参数进行优化设计,利用差分进化算法的全局寻优能力对ANFIS网络前件参数进行优化,而网络的结论参数采用最小二乘法优化,混合学习算法提高了网络参数辨识的收敛速度和系统的全局收敛性,仿真实验结果表明了该方法的有效性。     

基于EM-EKF算法的RBF-AR模型参数估计

为了利用EKF（extended Kalman filter）算法对RBF-AR（radial basis function network-based autoregressive）模型进行参数估计,重构了RBF-AR模型的网络结构,将其变换成一种新型的广义径向基函数（radial basis function,RBF）神经网络.与典型三层RBF网络结构相比,该广义RBF网络增加了线性输出加权层.为了克服基于EKF神经网络学习算法由于噪声统计特性未知导致滤波发散或者滤波精度不高的问题,利用EM（expectation maximization）算法对RBF-AR模型噪声协方差矩阵进行估计.同时,通过EKF滤波实时估计RBF-AR模型参数（系统状态）,EKF平滑过程得到了更加准确的期望估计.仿真结果显示,该方法用在此变形的RBF-AR模型结构中是有效的,特别在信噪比低的情况下,估计效果比SNPOM（structured nonlinear parameter optimization method）方法好,而且还能估计出噪声方差.F检验显示了两方法估计得到的标准偏差有显著性差异.     

使用新混合模糊聚类算法的模糊系统建模方法

为了进一步提高模糊系统建立模型的精度,提出一种新的模糊系统算法ANFIS-HC-QPSO：采用一种混合型模糊聚类算法来对模糊系统的输入空间进行划分,每一个聚类通过高斯函数的拟合产生一个隶属度函数,即完成ANFIS系统的前件参数--隶属度函数参数的初始识别,通过具有量子行为的粒子群算法QPSO与最小二乘法优化前件参数,直至达到停机条件,最终得到ANFIS的前件及后件参数,从而得到满意的模糊系统模型。实验表明,AN-FIS-HC-QPSO算法与传统算法相比,能在只需较少模糊规则的前提下就使模糊系统达到更高的精度。     

强跟踪平方根容积卡尔曼滤波和 自回归模型融合的故障预测

为了解决非线性系统中不可测量参数的预测问题,提出一种带有次优渐消因子的强跟踪平方根容积卡尔曼滤波(STSCKF)和自回归(AR)模型相结合的故障预测方法.利用AR模型时间序列预测法预测未来时刻的测量值,将预测的测量值作为STSCKF的测量变量,从而将预测问题转化为滤波估计问题.STSCKF通过在预测误差方差阵的均方根中引入渐消因子调节滤波过程中的增益矩阵,克服了故障参数变化函数未知情况下普通SCKF跟踪故障参数缓慢甚至失效的局限性,使得STSCKF能较好地预测故障参数的发展趋势.连续搅拌反应釜(CSTR)仿真结果表明,STSCKF的预测精度高于普通SCKF和强跟踪无迹卡尔曼滤波(STUKF),验证了方法的有效性.     

Blind Source Separation of Postnonlinear Convolutive Mixture

In this paper, a novel solution is developed to solve blind source separation of postnonlinear convolutive mixtures. The proposed model extends the conventional linear instantaneous mixture model to include both convolutive mixing and postnonlinear distortion. The maximum-likelihood (ML) approach solution based on the expectation-maximization (EM) algorithm is developed to estimate the source signals and the parameters in the proposed nonlinear model. In the proposed solution, the sufficient statistics associated with the source signals are estimated in the E-step, while the model parameters are optimized through these statistics in the M-step. However, the complication resulted from the postnonlinear function associated with the mixture renders these statistics difficult to be formulated in a closed form and hence causes intractability in the parameter optimization. A computationally efficient algorithm is proposed which uses the extended Kalman smoother (EKS) to facilitate the E-step tractable and a set of self-updated polynomials is used as the nonlinearity estimator to facilitate closed form estimations of the parameters in the M-step. The theoretical foundation of the proposed solution has been rigorously developed and discussed in details. Both simulations and recorded speech signals have been carried out to verify the success and efficacy of the proposed algorithm. Remarkable improvement has been obtained when compared with the existing algorithms.     

Identification of chemical processes with irregular output sampling

In many chemical processes, variables which indicate product quality are infrequently and irregularly sampled. Often, the inter-sample behavior of these quality variables can be inferred from manipulated variables and other process variables which are measured frequently. When the quality variables are irregularly sampled, maximum likelihood estimation (MLE) of the model parameters can be performed using the expectation maximization (EM) approach. A state-space model identification procedure based on the EM algorithm yields a Kalman filter-based prediction–correction mechanism which can be used for optimal prediction of the quality variables. In this paper, we describe such a state-space model identification and estimation method and present the results of its application on simulation, laboratory-scale and industrial case studies.     

An input-output clustering approach to the synthesis of ANFIS networks

A useful neural network paradigm for the solution of function approximation problems is represented by adaptive neuro-fuzzy inference systems (ANFIS). Data driven procedures for the synthesis of ANFIS networks are typically based on clustering a training set of numerical samples of the unknown function to be approximated. Some serious drawbacks often affect the clustering algorithms adopted in this context, according to the particular data space where they are applied. To overcome such problems, we propose a new ANFIS synthesis procedure where clustering is applied in the joint input-output data space. Using this approach, it is possible to determine the consequent part of Sugeno first-order rules and therefore the hyperplanes characterizing the local structure of the function to be approximated. Successively, the fuzzy antecedent part of each rule is determined using a particular fuzzy min-max classifier, which is based on the adaptive resolution mechanism. The generalization capability of the resulting ANFIS architecture is optimized using a constructive procedure for the automatic determination of the optimal number of rules. Simulation tests and comparisons with respect to other neuro-fuzzy techniques are discussed in the paper, in order to assess the efficiency of the proposed approach.     

A hybrid genetic algorithm–adaptive network-based fuzzy inference system in prediction of wave parameters

An important issue in application of fuzzy inference systems (FISs) to a class of system identification problems such as prediction of wave parameters is to extract the structure and type of fuzzy if–then rules from an available input–output data set. In this paper, a hybrid genetic algorithm–adaptive network-based FIS (GA–ANFIS) model has been developed in which both clustering and rule base parameters are simultaneously optimized using GAs and artificial neural nets (ANNs). The parameters of a subtractive clustering method, by which the number and structure of fuzzy rules are controlled, are optimized by GAs within which ANFIS is called for tuning the parameters of rule base generated by GAs. The model has been applied in the prediction of wave parameters, i.e. wave significant height and peak spectral period, in a duration-limited condition in Lake Michigan. The data set of year 2001 has been used as training set and that of year 2004 as testing data. The results obtained by the proposed model are presented and analyzed. Results indicate that GA–ANFIS model is superior to ANFIS and Shore Protection Manual (SPM) methods in terms of their prediction accuracy.     

基于在线分裂合并EM算法的高斯混合模型分类方法*

为了解决传统高斯混合模型中期望值EM处理必须具备足够数量的样本才能开始训练的问题,提出了一种新的高斯混合模型在线增量训练算法。本算法在Ueda等人提出的Split-and-Merge EM方法基础上对分裂合并准则的计算进行了改进,能够有效避免陷入局部极值并减少奇异值出现的情况;通过引入时间序列参数提出了增量EM训练方法,能够实现增量式的期望最大化训练,从而能够逐样本在线更新GMM模型参数。对合成数据和实际语音识别应用的实验结果表明,本算法具有较好的运算效率和分类准确性。     

Prediction of entrance length for low Reynolds number flow in pipe using neuro-fuzzy inference system

This paper proposes an adaptive network fuzzy inference system (ANFIS) for the prediction of entrance length in pipe for low Reynolds number flow. After using the computational fluid dynamics (CFD) technique to establish the basic database under various working conditions, an efficient rule database and optimal distribution of membership function is constructed from the hybrid-learning algorithm of ANFIS. An experimental data set is obtained with Reynolds number, diameter of the pipe, and inlet velocity as input parameters and entrance length as output parameter. The input-output data set is used for training and validation of the proposed techniques. After validation, they are forwarded for the prediction of entrance length. The entrance length estimation results obtained by the model are compared with existing predictive models and are presented. The model performed quite satisfactory results with the actual and predicted entrance length values. The model can also be used for estimating entrance length on-line but the accuracy of the model depends upon the proper training and selection of data points.     

A sparse kernel algorithm for online time series data prediction

Kernel based methods have been widely applied for signal analysis and processing. In this paper, we propose a sparse kernel based algorithm for online time series prediction. In classical kernel methods, the kernel function number is very large which makes them of a high computational cost and only applicable for off-line or batch learning. In online learning settings, the learning system is updated when each training sample is obtained and it requires a higher computational speed. To make the kernel methods suitable for online learning, we propose a sparsification method based on the Hessian matrix of the system loss function to continuously examine the significance of the new training sample in order to select a sparse dictionary (support vector set). The Hessian matrix is equivalent to the correlation matrix of sample inputs in the kernel weight updating using the recursive least square (RLS) algorithm. This makes the algorithm able to be easily implemented with an affordable computational cost for real-time applications. Experimental results show the ability of the proposed algorithm for both real-world and artificial time series data forecasting and prediction.     

State estimation and parameter identification method for dual-rate system based on improved Kalman prediction

For the dual-rate system, such as the process of space teleoperation whose control signals is partly determined by delayed feedback states, the state values and system parameters are coupled and influenced each other, which are hard to be estimated simultaneously. In this paper, we propose a novel method for this problem. Firstly, considering the asynchronism of the input and output sampling signals, an auxiliary model is modeled as a medium to the state and output functions. Secondly, the Kalman prediction algorithm is improved to estimate the state values at output signals of the dual-rate system. The general step is using the output estimated errors in original and auxiliary systems to modify the estimated state values of the auxiliary model, and then the unknown state values in original system is defined by the ones in auxiliary model. Based on improved Kalman algorithm and hierarchical identification algorithm, we present the detailed procedures of state estimation and parameter identification method for the dual-rate system. The processes of state estimation and parameter identification are calculated and modified alternately. Finally, the simulation results reveal that the state and parameters both approach to the real values and the state values converge faster than the parameters.     

Multi-environment model adaptation based on vector Taylor series for robust speech recognition

In this paper, we propose a multi-environment model adaptation method based on vector Taylor series (VTS) for robust speech recognition. In the training phase, the clean speech is contaminated with noise at different signal-to-noise ratio (SNR) levels to produce several types of noisy training speech and each type is used to obtain a noisy hidden Markov model (HMM) set. In the recognition phase, the HMM set which best matches the testing environment is selected, and further adjusted to reduce the environmental mismatch by the VTS-based model adaptation method. In the proposed method, the VTS approximation based on noisy training speech is given and the testing noise parameters are estimated from the noisy testing speech using the expectation-maximization (EM) algorithm. The experimental results indicate that the proposed multi-environment model adaptation method can significantly improve the performance of speech recognizers and outperforms the traditional model adaptation method and the linear regression-based multi-environment method.     

A Generalized Ellipsoidal Basis Function Based Online Self-constructing Fuzzy Neural Network

In this paper, we propose a Generalized ellipsoidal basis function based online self-constructing fuzzy neural network (GEBF-OSFNN) which extends the ellipsoidal basis function (EBF)-based fuzzy neural networks (FNNs) by permitting input variables to be modeled by dissymmetrical Gaussian functions (DGFs). Due to the flexibility and dissymmetry of left and right widths of the DGF, the partitioning made by DGFs in the input space is more flexible and more interpretable, and therefore results in a parsimonious FNN with high performance under the online learning algorithm. The geometric growing criteria and the error reduction ratio (ERR) method are used as growing and pruning strategies respectively to realize the structure learning algorithm which implements an optimal and compact network structure. The GEBF-OSFNN starts with no hidden neurons and does not need to partition the input space a priori. In addition, all free parameters in premises and consequents are adjusted online based on the ε-completeness of fuzzy rules and the linear least square (LLS) approach, respectively. The performance of the GEBF-OSFNN paradigm is compared with other well-known algorithms like RAN, RANEKF, MRAN, ANFIS, OLS, RBF-AFS, DFNN, GDFNN GGAP-RBF, OS-ELM, SOFNN and FAOS-PFNN, etc., on various benchmark problems in the areas of function approximation, nonlinear dynamic system identification, chaotic time-series prediction and real-world benchmark problems. Simulation results demonstrate that the proposed GEBF-OSFNN approach can facilitate a more powerful and more parsimonious FNN with better performance of approximation and generalization.     

Visual tracking using the Earth Mover's Distance between Gaussian mixtures and Kalman filtering

In this paper, we demonstrate how the differential Earth Mover's Distance (EMD) may be used for visual tracking in synergy with Gaussian mixtures models (GMM). According to our model, motion between adjacent frames results in variations of the mixing proportions of the Gaussian components representing the object to be tracked. These variations are computed in closed form by minimizing the differential EMD between Gaussian mixtures, yielding a very fast algorithm with high accuracy, without recurring to the EM algorithm in each frame. Moreover, we also propose a framework to handle occlusions, where the prediction for the object's location is forwarded to an adaptive Kalman filter whose parameters are estimated on line by the motion model already observed. Experimental results show significant improvement in tracking performance in the presence of occlusion.