Memetic evolutionary training for recurrent neural networks: an application to time-series prediction

Abstract: Artificial neural networks are bio-inspired mathematical models that have been widely used to solve complex problems. The training of a neural network is an important issue to deal with, since traditional gradient-based algorithms become easily trapped in local optimal solutions, therefore increasing the time taken in the experimental step. This problem is greater in recurrent neural networks, where the gradient propagation across the recurrence makes the training difficult for long-term dependences. On the other hand, evolutionary algorithms are search and optimization techniques which have been proved to solve many problems effectively. In the case of recurrent neural networks, the training using evolutionary algorithms has provided promising results. In this work, we propose two hybrid evolutionary algorithms as an alternative to improve the training of dynamic recurrent neural networks. The experimental section makes a comparative study of the algorithms proposed, to train Elman recurrent neural networks in time-series prediction problems.     

Multiple-valued feedback and recurrent correlation neural networks

In this paper we consider two kinds of neural networks in which the activation function of each neuron is a multiple valued, piecewised-constant function. The main advantages of the proposed models are that they can store patterns with different grey levels, and that they can store binary patterns with much fewer neurons than the existing models. We prove theoretically the convergence property of the proposed models. Different synthesis methods are developed to guarantee the storage of desired patterns as asymptotic equilibria. Simulation results confirm the effectiveness of the new models.     

Forecasting wind speed using empirical mode decomposition and Elman neural network

Because of the chaotic nature and intrinsic complexity of wind speed, it is difficult to describe the moving tendency of wind speed and accurately forecast it. In our study, a novel EMD–ENN approach, a hybrid of empirical mode decomposition (EMD) and Elman neural network (ENN), is proposed to forecast wind speed. First, the original wind speed datasets are decomposed into a collection of intrinsic mode functions (IMFs) and a residue by EMD, yielding relatively stationary sub-series that can be readily modeled by neural networks. Second, both IMF components and residue are applied to establish the corresponding ENN models. Then, each sub-series is predicted using the corresponding ENN. Finally, the prediction values of the original wind speed datasets are calculated by the sum of the forecasting values of every sub-series. Moreover, in the ENN modeling process, the neuron number of the input layer is determined by a partial autocorrelation function. Four prediction cases of wind speed are used to test the performance of the proposed hybrid approach. Compared with the persistent model, back-propagation neural network, and ENN, the simulation results show that the proposed EMD–ENN model consistently has the minimum statistical error of the mean absolute error, mean square error, and mean absolute percentage error. Thus, it is concluded that the proposed approach is suitable for wind speed prediction.     

服务器性能预测的递归神经元网络方法

正确有效地预测服务器性能负载,是计算机系统性能管理系统的一个重要环节。通常,传统的预测方法有最小二乘、二次指数平滑法等,但这些模型往往不能很好地捕捉服务器性能负载数据的时序关系。利用基于局部回归的递归神经网络（RNN）,采用改进的RPROP学习算法进行服务器性能负载的预测。并与传统的二次指数平滑法相比较,实验结果证明,RNN得到的预测结果要比二次指数平滑法高出5个百分点以上,并且有较强的预测能力,可以预测较长周期的数据。     

Pressure prediction on a variable-speed pump controlled hydraulic system using structured recurrent neural networks

This paper presents a study to predict the pressures in the cylinder chambers of a variable-speed pump controlled hydraulic system using structured recurrent neural network topologies where the rotational speed of the pumps, the position and the average velocity of the hydraulic actuator are used as their inputs. The paper elaborates the properties of such networks in extended time periods through detailed simulation- and experimental studies where black-box modeling approaches generally fail to yield acceptable performance. As alternative estimation techniques, both linear- and extended Kalman filters are considered in this paper. The estimation properties of the devised network models are comparatively evaluated and their potential application areas are discussed in detail.     

DeepEC: An error correction framework for dose prediction and organ segmentation using deep neural networks

Radiotherapy is an indispensable part of adjuvant therapy for cancer that improves local control, overall survival, and the opportunity for good quality of life. Organ delineation and dose plan design are the key steps in the treatment. Organ delineation controls the area of radiotherapy and dose planning controls its intensity. However, both tasks are time-consuming, exhausting, and subjective, and automated methods are desirable. Although automated methods have been studied, the previous studies either focus on organ segmentation or dose prediction, without considering them from a holistic perspective. In this paper, we treat organ segmentation and dose prediction as similar tasks, and propose an error correction framework to improve their performance based on the same mechanism. The proposed error correction framework consists of a prediction network and a calibration network. The biggest difference between our framework and previous studies is that the state-of-the-art networks can be used as a prediction network or calibration network, and then the performance can be improved by the error correction mechanism. To evaluate the framework, we conducted a series of experiments on dose prediction and organ segmentation. These experimental results show that the framework is superior to other state-of-the-art methods in both tasks.     

基于递归神经网络的网络流量组合预测模型

为了提高网络流量的预测精度,提出了一种基于Elman递归神经网络,小波和自回归的网络流量组合预测模型.对流量时间序列进行小波分解,得到小波变换尺度系数序列和小波系数序列,对具有平稳特征的尺度系数序列用AR模型进行预测:而对体现了网络流量非线性、非平稳特性的小波系数序列使用Elman递归神经网络进行预测,最后通过Mallat算法重构得到网络流量的预测值.     

Multiscale prediction of wind speed and output power for the wind farm

This paper focuses on establishing the multiscale prediction models for wind speed and power in wind farm by the average wind speed collected from the history records. Each type of the models is built with different time scales and by different approaches. There are three types of them that a short-term model for a day ahead is based on the least squares support vector machine (LSSVM), a medium-term model for a month ahead is on the combination of LSSVM and wavelet transform (WT), and a long-term model for a year ahead is on the empirical mode decomposition (EMD) and recursive least square (RLS) approaches. The simulation studies show that the average value of the mean absolute percentage error (MAPE) is 4.91%, 6.57% and 16.25% for the short-term, the medium-term and the long-term prediction, respectively. The predicted data also can be used to calculate the predictive values of output power for the wind farm in different time scales, combined with the generator’s power characteristic, meteorologic factors and unit efficiency under various operating conditions.     

基于构造性神经网络的时间序列混合预测模型 *

针对传统时间序列预测模型不适应非线性预测而适应非线性预测的BP算法存在收敛速度慢,且容易陷入局部极小等问题,提出一种基于构造性神经网络的时间序列混合预测模型。采用构造性神经网络模型（覆盖算法）得出的类别值对统计时间序列模型的预测值进行修正,建立一种同时考虑时间序列自身周期变化和外生变量因子对时间序列未来变化趋势影响的混合预测模型,涵盖了实际问题的线性和非线性两方面,提高了预测精度。将该模型应用到粮食产量的预测中,取得了较好的预测效果。     

基于构造性神经网络的时间序列混合预测模型 *

针对传统时间序列预测模型不适应非线性预测而适应非线性预测的 BP算法存在收敛速度慢 ,且容易陷入局部极小等问题 ,提出一种基于构造性神经网络的时间序列混合预测模型。采用构造性神经网络模型 (覆盖算法 )得出的类别值对统计时间序列模型的预测值进行修正 ,建立一种同时考虑时间序列自身周期变化和外生变量因子对时间序列未来变化趋势影响的混合预测模型 ,涵盖了实际问题的线性和非线性两方面 ,提高了预测精度。将该模型应用到粮食产量的预测中 ,取得了较好的预测效果。     

Towards designing modular recurrent neural networks in learning protein secondary structures

Precise prediction of protein secondary structures from the associated amino acids sequence is of great importance in bioinformatics and yet a challenging task for machine learning algorithms. As a major step toward predicting the ultimate three dimensional structures, the secondary structure assignment specifies the protein function. Considering a multilayer perceptron neural network, pruned for optimum size of hidden layers, as the reference network, advanced kinds of recurrent neural network (RNN) are devised in this article to enhance the secondary structure prediction. To better model the strong correlations between secondary structure elements, types of modular reciprocal recurrent neural networks (MRR-NN) are examined. Additionally, to take into account the long-range interactions between amino acids in formation of the secondary structure, bidirectional RNN are investigated. A multilayer bidirectional recurrent neural network (MBR-NN) is finally applied to capture the predominant long-term dependencies. Eventually, a modular prediction system based on the interactive combination of the MRR-NN and MBR-NN boosts the percentage accuracy (Q₃) up to 76.91% and augments the segment overlap (SOV) up to 68.13% when tested on the PSIPRED dataset. The coupling effects of the secondary structure types as well as the sequential information of amino acids along the protein chain can be well cast by the integration of the MRR-NN and the MBR-NN.     

SOM time series clustering and prediction with recurrent neural networks

Local models for regression have been the focus of a great deal of attention in the recent years. They have been proven to be more efficient than global models especially when dealing with chaotic time series. Many models have been proposed to cluster time series and they have been combined with several predictors. This paper presents an extension for recurrent neural networks applied to local models and a discussion about the obtained results.     

共溶剂-SC CO_2中固体溶解度的小波神经网络关联与预测

为了更好地关联和预测共溶剂-超临界(SC)CO2中固体的溶解度,本文采用经局部动量法和自适应理论优化的小波神经网络模型(WNN),分别以共溶剂的溶剂参数α、β、π*和溶解度参数δd、δp、δh为影响因素,关联了4种共溶剂-SC CO2体系中萘普生的溶解度,以预测其在乙醇-SC CO2中的溶解度.其关联误差AARD分别为1.94%和2.12%;其预测误差AARD分别为8.14%和30.32%.以上结果表明共溶剂的溶剂参数α、β、π*是共溶剂-SC CO2中固体溶解度的主要影响因素;优化的WNN模型能较好地关联和预测共溶剂-SC CO2中固体的溶解度.     

Adjoint EKF learning in recurrent neural networks for nonlinear active noise control

In this paper, active noise control using recurrent neural networks is addressed. A new learning algorithm for recurrent neural networks based on Adjoint Extended Kalman Filter is developed for active noise control. The overall control structure for active noise control is constructed using two recurrent neural networks: the first neural network is used to model secondary path of active noise control while the second one is employed to generate control signal. Real-time experiment of the proposed algorithm using digital signal processor is carried-out to show the effectiveness of the method.     

基于马尔科夫链的传感器网络空间相关性数据预测算法

如何降低计算复杂度的同时提高数据预测的精度,是传感器网络数据预测研究中亟需解决的问题之一。针对目前主流的时间相关性数据预测算法在数据波动大时预测精度低的问题,本文引入Delaunay三角形邻近图来度量网络中监测数据的空间相关性,并提出基于马尔科夫链的空间相关性数据预测算法。实验表明,该算法可以在数据波动不规律的情况下提高数据预测精度,减少网络中数据传输量。     

Exponential Lagrange stability for impulses in discrete-time delayed recurrent neural networks

This paper focuses on the problem of exponential stability in the sense of Lagrange for impulses in discrete-time delayed recurrent neural networks. By establishing a delayed impulsive discrete inequality and a novel difference inequality, combining with inequality techniques, some novel sufficient conditions are obtained to ensure exponential Lagrange stability for impulses in discrete-time delayed recurrent neural networks. Meanwhile, exponentially convergent scope of neural network is given. Finally, several numerical simulations are given to demonstrate the effectiveness of our results.     

基于神经网络的卫星钟差预报研究

针对卫星钟差的数学预测补偿问题,研究基于径向基函数神经网络的卫星钟差预报方法.以国际全球定位系统服务机构数据中心网站公布的卫星钟差数据作为输入样本,利用径向基函数神经网络进行学习训练,进行卫星钟差预报,将结果与线性插值法和多项式插值法以及数据中心发布的精密卫星钟差数据做比较,分析卫星钟差的预报精度.仿真结果表明,RBF神经网络逼近非线性函数的能力强、收敛速度快,卫星钟差预报精度高、残差波动小、拟合平滑度好.     

Dynamic classification of ballistic missiles using neural networks and hidden Markov models

This paper addresses dynamic classification of different ranges of ballistic missiles (BM) for air defense application based on kinematic attributes acquired by radars for taking appropriate measures to intercept them. The problem of dynamic classification is formulated using real-time neural network (RTNN) and hidden Markov model (HMM). The idea behind these algorithms is to calculate the output in one pass rather than training and computing over large number of iterations. Besides, to meet the conflicting requirements of classifying small as well as long-range trajectories, we are also proposing a formulation for partitioning the trajectory by using moving window concept. This concept allows us to use parameters in localized frame which helps in handling wide-range of trajectories to fit into the same network. These algorithms are evaluated using the simulated data generated from 6 degree-of-freedom (6DOF) mathematical model, which models missile trajectories. Experimental results show that both the networks are classifying above 95% with real-time neural network outperforming HMM in terms of time of computation on same data. The small classification time enables the use of real-time classification neural network in complex scenario of multi-radar, multi-target engagement by interceptor missiles. To the best of our knowledge this is the first time an attempt is made to classify ballistic missiles using RTNN and HMM.     

递归pi-sigma神经网络的带惩罚项的梯度算法分析

传统的梯度算法存在收敛速度过慢的问题,针对这个问题,提出一种将惩罚项加到传统误差函数的梯度算法以训练递归pi-sigma神经网络,算法不仅提高了神经网络的泛化能力,而且克服了因网络初始权值选取过小而导致的收敛速度过慢的问题,相比不带惩罚项的梯度算法提高了收敛速度。从理论上分析了带惩罚项的梯度算法的收敛性,并通过实验验证了算法的有效性。     

一种新的对角回归神经网络快速学习算法

提出一种新的动态对角回归神经网络学习算法－局部动态误差反传算法（LDBP），该算法定义了一种新的局部均方差函数，并为回归单元建立一种新的学习结构。如果估计出各层的期望输出值，多层回归网络便可分解成一组自适应单元（Adaline），而每个单元可通过二次优化方法进行训练。采用可在有限步人找出全局最优解的共轭梯度法（CG）进行寻优。由于学习过程采用超线性搜索，大大减少了循环步数和计算时间。