Identification of nonlinear systems with outliers using wavelet neural networks based on annealing dynamical learning algorithm

This paper presents an annealing dynamical learning algorithm (ADLA) to train wavelet neural networks (WNNs) for identifying nonlinear systems with outliers. In ADLA–WNNs, wavelet-based support vector regression (WSVR) is adopted to determine the initial translation and dilation of a wavelet kernel and the weights of WNNs due to the similarity between WSVR and WNNs. After initialization, ADLA with nonlinear time-varying learning rates is applied to train the WNNs. In the ADLA, the determination of the learning rates would be a key work for the trade-off between stability and speed of convergence. A computationally efficient optimization method, particle swarm optimization (PSO), is adopted to find the optimal learning rates to overcome the stagnation in the training procedure of WNNs. Due to the advantages of WSVR and ADLA (WSVR–ADLA), the WSVR-based ADLA–WNNs (WSVR–ADLA–WNNs) can robust against outliers and achieve the promising efficiency of system identifications. Three examples are simulated to confirm the performance of the proposed algorithm. From the simulated results, the feasibility and superiority of the proposed WSVR–ADLA–WNNs for identifying nonlinear systems with artificial outliers are verified.     

ARRBFNs with SVR for prediction of chaotic time series with outliers

In this article, annealing robust radial basis function networks (ARRBFNs), which consist of a radial basis function network and a support vector regression (SVR), and an annealing robust learning algorithm (ARLA) are proposed for the prediction of chaotic time series with outliers. In order to overcome the initial structural problems of the proposed neural networks, the SVR is utilized to determine the number of hidden nodes, the initial parameters of the kernel, and the initial weights for the proposed ARRBFNs. Then the ARLA that can conquer the outliers is applied to tune the parameters of the kernel and the weights in the proposed ARRBFNs under the initial structure with SVR. The simulation results of Mackey-Glass time series show that the proposed approach with different SVRs can cope with outliers and give a fast learning speed. The results of the simulation are also given to demonstrate the validity of proposed method for chaotic time series with outliers.     

A fast identification algorithm with outliers under Box-Cox transformation-based annealing robust radial basis function networks

In this article, a Box-Cox transformation-based annealing robust radial basis function networks (ARRBFNs) is proposed for an identification algorithm with outliers. Firstly, a fixed Box-Cox transformation-based ARRBFN model with support vector regression (SVR) is derived to determine the initial structure. Secondly, the results of the SVR are used as the initial structure in the fixed Box-Cox transformation-based ARRBFNs for the identification algorithm with outliers. At the same time, an annealing robust learning algorithm (ARLA) is used as the learning algorithm for the fixed Box-Cox transformation-based ARRBFNs, and applied to adjust the parameters and weights. Hence, the fixed Box-Cox transformation-based ARRBFNs with an ARLA have a fast convergence speed for an identification algorithm with outliers. Finally, the proposed algorithm and its efficacy are demonstrated with an illustrative example in comparison with Box-Cox transformation-based radial basis function networks.     

Optimizing wavelet neural networks using modified cuckoo search for multi-step ahead chaotic time series prediction

Determining the optimal number of hidden nodes and their proper initial locations are essentially crucial before the wavelet neural networks (WNNs) start their learning process. In this paper, a novel strategy known as the modified cuckoo search algorithm (MCSA), is proposed for WNNs initialization in order to improve its generalization performance. The MCSA begins with an initial population of cuckoo eggs, which represent the translation vectors of the wavelet hidden nodes, and subsequently refines their locations by imitating the breeding mechanism of cuckoos. The resulting solutions from the MCSA are then used as the initial translation vectors for the WNNs. The feasibility of the proposed method is evaluated by forecasting a benchmark chaotic time series, and its superior prediction accuracy compared with that of conventional WNNs demonstrates its potential benefit.     

基于小波神经网络的混沌时间序列分析与相空间重构

探讨了小波神经网络在混沌时间序列分析与相空间重构中的应用,通过混沌时间序列单步预测与多步预测的例子,比较了小波神经网络与MLP的逼近和收敛性能,对最近提出的一种多分辨率学习策略进行了改进,利用连续3次样条小和正交Daubechies小波代替Haar小波对时间序列做小波分解;用改进的学习算法训练网络并应用到混沌序列相空间重构中,实验结果表明,小波神经网络比MLP和ARMA模型具有更强大的逼近能力,因而十分适合应用于时间序列分析中;多分辨率学习算法可作为分析复杂混沌时间序列的一种重要工具。     

利用矢量基学习和自适应迭代算法改进LSSVR

针对最小二乘支持向量回归缺乏传统SVR的稀疏性和鲁棒性等问题,综合矢量基学习和自适应迭代算法的优势,提出了一种改进的加权最小二乘支持向量回归算法（LSSVR）。该算法通过引入用矢量基学习和自适应迭代相结合的方式得到一个小的支持向量集,可以避免递推时可能出现的误差积累问题,有效提高算法的稀疏性和稳定性;同时采用加权方法确定权值系数以减小训练样本中非高斯噪声的影响。实验结果表明,改进的LSSVR具有较好的鲁棒性、支持向量稀疏性和动态建模实时性。     

Robust least squares support vector machine based on recursive outlier elimination

To achieve robust estimation for noisy data set, a recursive outlier elimination-based least squares support vector machine (ROELS-SVM) algorithm is proposed in this paper. In this algorithm, statistical information from the error variables of least squares support vector machine is recursively learned and a criterion derived from robust linear regression is employed for outlier elimination. Besides, decremental learning technique is implemented in the recursive training–eliminating stage, which ensures that the outliers are eliminated with low computational cost. The proposed algorithm is compared with re-weighted least squares support vector machine on multiple data sets and the results demonstrate the remarkably robust performance of the ROELS-SVM.     

Optimization of wavelet neural networks with the firefly algorithm for approximation problems

The effectiveness of swarm intelligence has been proven to be at the heart of various optimization problems. In this study, a recently developed nature-inspired algorithm, specifically the firefly algorithm (FA), is integrated in the learning strategy of wavelet neural networks (WNNs). The FA, which systematically optimizes the initial location of the translation parameters for WNNs, has reduced the number of hidden nodes while simultaneously improved the generalization capability of WNNs significantly. The applicability of the proposed model was demonstrated through empirical simulations for function approximation study, with both synthetic and real-world data. Performance assessment demonstrated its enhancement over the K-means clustering and random initialization approaches, as well as to the other neural network models reported in the literature, whereby a noteworthy decrease in the approximation error was observed.

     

Wavelet basis function neural networks for sequential learning.

In this letter, we develop the wavelet basis function neural networks (WBFNNs). It is analogous to radial basis function neural networks (RBFNNs) and to wavelet neural networks (WNNs). In WBFNNs, both the scaling function and the wavelet function of a multiresolution approximation (MRA) are adopted as the basis for approximating functions. A sequential learning algorithm for WBFNNs is presented and compared to the sequential learning algorithm of RBFNNs. Experimental results show that WBFNNs have better generalization property and require shorter training time than RBFNNs.     

基于改进的LSSVM辨识动态非线性时变系统

针对非线性时变系统难以辨识的问题,提出了一种基于改进最小二乘支持向量机的辨识新方法。该方法在加权最小二乘支持向量机的基础上,引入用矢量基学习和自适应迭代相结合的方式得到一个小的支持向量,同时采用加权方法确定权值系数以减小训练样本中非高斯噪声的影响。通过对动态非线性时变系统的仿真,结果表明该算法具有较好的鲁棒性、支持向量稀疏性和动态建模实时性。     

CPBUM neural networks for modeling with outliers and noise

In this study, CPBUM neural networks with annealing robust learning algorithm (ARLA) are proposed to improve the problems of conventional neural networks for modeling with outliers and noise. In general, the obtained training data in the real applications maybe contain the outliers and noise. Although the CPBUM neural networks have fast convergent speed, these are difficult to deal with outliers and noise. Hence, the robust property must be enhanced for the CPBUM neural networks. Additionally, the ARLA can be overcome the problems of initialization and cut-off points in the traditional robust learning algorithm and deal with the model with outliers and noise. In this study, the ARLA is used as the learning algorithm to adjust the weights of the CPBUM neural networks. It tunes out that the CPBUM neural networks with the ARLA have fast convergent speed and robust against outliers and noise than the conventional neural networks with robust mechanism. Simulation results are provided to show the validity and applicability of the proposed neural networks.     

Linear programming support vector regression with wavelet kernel: A new approach to nonlinear dynamical systems identification

Wavelet theory has a profound impact on signal processing as it offers a rigorous mathematical framework to the treatment of multiresolution problems. The combination of soft computing and wavelet theory has led to a number of new techniques. On the other hand, as a new generation of learning algorithms, support vector regression (SVR) was developed by Vapnik et al. recently, in which ?-insensitive loss function was defined as a trade-off between the robust loss function of Huber and one that enables sparsity within the SVs. The use of support vector kernel expansion also provides us a potential avenue to represent nonlinear dynamical systems and underpin advanced analysis. However, for the support vector regression with the standard quadratic programming technique, the implementation is computationally expensive and sufficient model sparsity cannot be guaranteed. In this article, from the perspective of model sparsity, the linear programming support vector regression (LP-SVR) with wavelet kernel was proposed, and the connection between LP-SVR with wavelet kernel and wavelet networks was analyzed. In particular, the potential of the LP-SVR for nonlinear dynamical system identification was investigated.     

A robust learning algorithm based on support vector regression and robust fuzzy cerebellar model articulation controller

For real-world applications, the obtained data are always subject to noise or outliers. The learning mechanism of cerebellar model articulation controller (CMAC), a neurological model, is to imitate the cerebellum of human being. CMAC has an attractive property of learning speed in which a small subset addressed by the input space determines output instantaneously. For fuzzy cerebellar model articulation controller (FCMAC), the concept of fuzzy is incorporated into CMAC to improve the accuracy problem. However, the distributions of errors into the addressed hypercubes may cause unacceptable learning performance for input data with noise or outliers. For robust fuzzy cerebellar model articulation controller (RFCMAC), the robust learning of M-estimator can be embedded into FCMAC to degrade noise or outliers. Meanwhile, support vector machine (SVR) is a machine learning theory based algorithm which has been applied successfully to a number of regression problems when noise or outliers exist. Unfortunately, the practical application of SVR is limited to defining a set of parameters for obtaining admirable performance by the user. In this paper, a robust learning algorithm based on support SVR and RFCMAC is proposed. The proposed algorithm has both the advantage of SVR, the ability to avoid corruption effects, and the advantage of RFCMAC, the ability to obtain attractive properties of learning performance and to increase accurate approximation. Additionally, particle swarm optimization (PSO) is applied to obtain the best parameters setting for SVR. From simulation results, it shows that the proposed algorithm outperforms other algorithms.     

Hybrid robust support vector machines for regression with outliers

In this study, a hybrid robust support vector machine for regression is proposed to deal with training data sets with outliers. The proposed approach consists of two stages of strategies. The first stage is for data preprocessing and a support vector machine for regression is used to filter out outliers in the training data set. Since the outliers in the training data set are removed, the concept of robust statistic is not needed for reducing the outliers’ effects in the later stage. Then, the training data set except for outliers, called as the reduced training data set, is directly used in training the non-robust least squares support vector machines for regression (LS-SVMR) or the non-robust support vector regression networks (SVRNs) in the second stage. Consequently, the learning mechanism of the proposed approach is much easier than that of the robust support vector regression networks (RSVRNs) approach and of the weighted LS-SVMR approach. Based on the simulation results, the performance of the proposed approach with non-robust LS-SVMR is superior to the weighted LS-SVMR approach when the outliers exist. Moreover, the performance of the proposed approach with non-robust SVRNs is also superior to the RSVRNs approach.     

最小二乘Littlewood-Paley小波支持向量机

基于小波分解理论和支持向量机核函数的条件，提出了一种多维允许支持向量核函数——Littlewood-Paley小波核函数．该核函数不仅具有平移正交性，而且可以以其正交性逼近二次可积空间上的任意曲线，从而提升了支持向量机的泛化性能．在Littlewood-Paley小波函数作为支持向量核函数的基础上，提出了最小二乘Littlewood-Paley小波支持向量机(LS-LPWSVM)．实验结果表明，LS-LPWSVM在同等条件下比最小二乘支持向量机的学习精度要高，因而更适用于复杂函数的学习问题．     

The annealing robust backpropagation (ARBP) learning algorithm

Multilayer feedforward neural networks are often referred to as universal approximators. Nevertheless, if the used training data are corrupted by large noise, such as outliers, traditional backpropagation learning schemes may not always come up with acceptable performance. Even though various robust learning algorithms have been proposed in the literature, those approaches still suffer from the initialization problem. In those robust learning algorithms, the so-called M-estimator is employed. For the M-estimation type of learning algorithms, the loss function is used to play the role in discriminating against outliers from the majority by degrading the effects of those outliers in learning. However, the loss function used in those algorithms may not correctly discriminate against those outliers. In the paper, the annealing robust backpropagation learning algorithm (ARBP) that adopts the annealing concept into the robust learning algorithms is proposed to deal with the problem of modeling under the existence of outliers. The proposed algorithm has been employed in various examples. Those results all demonstrated the superiority over other robust learning algorithms independent of outliers. In the paper, not only is the annealing concept adopted into the robust learning algorithms but also the annealing schedule k/t was found experimentally to achieve the best performance among other annealing schedules, where k is a constant and t is the epoch number.     

Wavelet twin support vector machine

Twin support vector machine (TWSVM) is a research hot spot in the field of machine learning in recent years. Although its performance is better than traditional support vector machine (SVM), the kernel selection problem still affects the performance of TWSVM directly. Wavelet analysis has the characteristics of multivariate interpolation and sparse change, and it is suitable for the analysis of local signals and the detection of transient signals. The wavelet kernel function based on wavelet analysis can approximate any nonlinear functions. Based on the wavelet kernel features and the kernel function selection problem, wavelet twin support vector machine (WTWSVM) is proposed by this paper. It introduces the wavelet kernel function into TWSVM to make the combination of wavelet analysis techniques and TWSVM come true. The experimental results indicate that WTWSVM is feasible, and it improves the classification accuracy and generalization ability of TWSVM significantly.     

高斯小波支持向量机的研究

证明了偶数阶高斯小波函数满足支持向量机的平移不变核函数条件．应用小波核函数建立了相应的高斯小波支持向量机,并且使用云遗传算法对支持向量机及其核函数的参数进行优化．用该算法与常用的高斯核和Morlet小波核支持向量机进行对比实验．通过对非线性函数的逼近和电力系统短期负荷的预测,验证了该算法的有效性和优越性,表明其具有一定的实用价值．     

求解非线性回归问题的Newton算法

针对大规模非线性回归问题,提出基于静态储备池的Newton算法.利用储备池搭建高维特征空间,将原始问题转化成与储备池维数相关的线性支持向量回归问题,并应用Newton算法求解.鲁棒损失函数的应用可抑制异常点对预测结果的干扰.通过与SVR(Support Vector Regression)及储备池Tikhonov正则化方法比较,验证了所提方法的快速性、较高的预测精度和较好的鲁棒性.