State estimation in MIMO nonlinear systems subject to unknown deadzones using recurrent neural networks

This paper deals with the problem of state observation by means of a continuous-time recurrent neural network for a broad class of MIMO unknown nonlinear systems subject to unknown but bounded disturbances and with an unknown deadzone at each input. With respect to previous works, the main contribution of this study is twofold. On the one hand, the need of a matrix Riccati equation is conveniently avoided; in this way, the design process is considerably simplified. On the other hand, a faster convergence is carried out. Specifically, the exponential convergence of Euclidean norm of the observation error to a bounded zone is guaranteed. Likewise, the weights are shown to be bounded. The main tool to prove these results is Lyapunov-like analysis. A numerical example confirms the feasibility of our proposal.     

Early event detection in a deep-learning driven quality prediction model for ultrasonic welding

A goal in ultrasonic welding (USW) process monitoring is to accurately predict quality outcomes based on monitored signals. However, in most cases, knowing only that the USW process has failed is insufficient. Modern process automation should assess signal information and intercede to rectify process problems. Identification of when a process signal deviates from an acceptable final quality outcome, i.e., the time at which an abnormal event starts, facilitates control action or root cause analysis to bring it back to compliance. A long short-term memory (LSTM) recurrent neural network is proposed to monitor USW and other time-series signals and identify this point. This deep neural network is trained to classify quality outcomes from continuous signals. The process monitoring signals and their sampling time are divided into finite segments as input to this network. The time segment at which the process signal first converges to the final quality class prediction is identified using cross-entropy of the classification probabilities. This procedure is demonstrated using USW quality monitoring algorithms and robot motion failure detection. The examples show an LSTM network not only provides high accuracy for USW quality prediction, but also that the time of classification convergence is consistent with variance observed in USW weld quality factors. Moreover, classification convergence time was shown to be associated to specific robot motion failures, useful as input to adaptive learning. This work realizes deep-learning driven quality prediction and early event detection for quality classification problems, and provides the information necessary for adaptive control algorithms.     

Stock Price Forecasting: An Echo State Network Approach

Forecasting stock prices using deep learning models suffers from problems such as low accuracy, slow convergence, and complex network structures. This study developed an echo state network (ESN) model to mitigate such problems. We compared our ESN with a long short-term memory (LSTM) network by forecasting the stock data of Kweichow Moutai, a leading enterprise in China’s liquor industry. By analyzing data for 120, 240, and 300 days, we generated forecast data for the next 40, 80, and 100 days, respectively, using both ESN and LSTM. In terms of accuracy, ESN had the unique advantage of capturing nonlinear data. Mean absolute error (MAE) was used to present the accuracy results. The MAEs of the data forecast by ESN were 0.024, 0.024, and 0.025, which were, respectively, 0.065, 0.007, and 0.009 less than those of LSTM. In terms of convergence, ESN has a reservoir state-space structure, which makes it perform faster than other models. Root-mean-square error (RMSE) was used to present the convergence time. In our experiment, the RMSEs of ESN were 0.22, 0.27, and 0.26, which were, respectively, 0.08, 0.01, and 0.12 less than those of LSTM. In terms of network structure, ESN consists only of input, reservoir, and output spaces, making it a much simpler model than the others. The proposed ESN was found to be an effective model that, compared to others, converges faster, forecasts more accurately, and builds time-series analyses more easily.     

Forecasting foreign exchange rates with an improved back-propagation learning algorithm with adaptive smoothing momentum terms

The slow convergence of back-propagation neural network (BPNN) has become a challenge in data-mining and knowledge discovery applications due to the drawbacks of the gradient descent (GD) optimization method, which is widely adopted in BPNN learning. To solve this problem, some standard optimization techniques such as conjugate-gradient and Newton method have been proposed to improve the convergence rate of BP learning algorithm. This paper presents a heuristic method that adds an adaptive smoothing momentum term to original BP learning algorithm to speedup the convergence. In this improved BP learning algorithm, adaptive smoothing technique is used to adjust the momentums of weight updating formula automatically in terms of “3 σ limits theory.” Using the adaptive smoothing momentum terms, the improved BP learning algorithm can make the network training and convergence process faster, and the network’s generalization performance stronger than the standard BP learning algorithm can do. In order to verify the effectiveness of the proposed BP learning algorithm, three typical foreign exchange rates, British pound (GBP), Euro (EUR), and Japanese yen (JPY), are chosen as the forecasting targets for illustration purpose. Experimental results from homogeneous algorithm comparisons reveal that the proposed BP learning algorithm outperforms the other comparable BP algorithms in performance and convergence rate. Furthermore, empirical results from heterogeneous model comparisons also show the effectiveness of the proposed BP learning algorithm.     

A shape-based adaptive segmentation of time-series using particle swarm optimization

The increasing size of large databases has motivated many researchers to develop methods to reduce the dimensionality of data so that their further analysis can be easier and faster. There are many techniques for time-series’ dimensionality reduction; however, majority of them need an input by the user such as the number of segments. In this paper, the segmentation problem is analyzed from the optimization point of view. A new approach for time-series’ segmentation based on Particle Swarm Optimization (PSO) is proposed which is highly adaptive to time-series’ shape and shape-based characteristics. The proposed approach, called Adaptive Particle Swarm Optimization Segmentation (APSOS), is tested on various datasets to demonstrate its effectiveness and efficiency. Experiments are conducted to show that APSOS is independent of user input parameters and the results indicate that the proposed approach outperforms common methods used for the time-series segmentation.     

Backpropagation of an Image Similarity Metric for Autoassociative Neural Networks

Autoassociative Neural Networks (AANNs) are most commonly used for image data compression. The goal of an AANN for image data is to have the network output be ‘similar’ to the input. Most of the research in this area use backpropagation training with Mean-Squared Error (MSE) as the optimisation criteria. This paper presents an alternative error function called the Visual Difference Predictor (VDP) based on concepts from the human-visual system. Using the VDP as the error function provides a criteria to train an AANN more efficiently, and results in faster convergence of the weights, while producing an output image perceived to be very similar by a human observer. Received: 02 December 1998, Received in revised form: 28 June 1999, Accepted: 05 August 1999     

A diagonal recurrent neural network-based hybrid direct adaptiveSPSA control system

A direct adaptive simultaneous perturbation stochastic approximation (DA SPSA) control system with a diagonal recurrent neural network (DRNN) controller is proposed. The DA SPSA control system with DRNN has simpler architecture and parameter vector size that is smaller than a feedforward neural network (FNN) controller. The simulation results show that it has a faster convergence rate than FNN controller. It results in a steady-state error and is sensitive to SPSA coefficients and termination condition. For trajectory control purpose, a hybrid control system scheme with a conventional PID controller is proposed     

A fast and efficient two-phase sequential learning algorithm for spatial architecture neural network

Spatial architecture neural network (SANN), which is inspired by the connecting mode of excitatory pyramidal neurons and inhibitory interneurons of neocortex, is a multilayer artificial neural network and has good learning accuracy and generalization ability when used in real applications. However, the backpropagation-based learning algorithm (named BP-SANN) may be time consumption and slow convergence. In this paper, a new fast and accurate two-phase sequential learning scheme for SANN is hereby introduced to guarantee the network performance. With this new learning approach (named SFSL-SANN), only the weights connecting to output neurons will be trained during the learning process. In the first phase, a least-squares method is applied to estimate the span-output-weight on the basis of the fixed randomly generated initialized weight values. The improved iterative learning algorithm is then used to learn the feedforward-output-weight in the second phase. Detailed effectiveness comparison of SFSL-SANN is done with BP-SANN and other popular neural network approaches on benchmark problems drawn from the classification, regression and time-series prediction applications. The results demonstrate that the SFSL-SANN is faster convergence and time-saving than BP-SANN, and produces better learning accuracy and generalization performance than other approaches.     

时序神经网络算法的短期风速预测研究

采用神经网络对风速进行短期预测,研究BP型短期风速预测网络中BP算法、BP网络构建以及网络训练方法.结合时间序列法和神经网络法提出了时序神经网络预测方法,对短期风速预测网络中输入变量数量和隐舍层节点数量的选择方法进行了探讨.仿真实验结果表明,时序神经网络法建立的网络,训练时间明显缩短,网络输出的预测值与真实的观察值之间...     

基于权值变化的BP神经网络自适应学习率改进研究

针对传统神经网络的学习率由人为经验性设定,存在学习率设置过大或过小,容易导致无法收敛或收敛速度慢的问题,本文提出基于权值变化的自适应学习率改进方法,改善传统神经网络学习率受人为经验因素影响的弊端,提高误差精度,并结合正态分布模型与梯度上升法,提高收敛速度.本文以BP神经网络为例,对比固定学习率的神经网络,应用经典XOR问题仿真验证,结果表明本文的改进神经网络具有更快的收敛速度和更小的误差.     

Learning as a nonlinear line of attraction in a recurrent neural network

A method to embed N dimensional, multi-valued patterns into an auto-associative memory represented as a nonlinear line of attraction in a fully connected recurrent neural network is presented in this paper. The curvature of the nonlinear attractor is defined by the Kth degree polynomial line which best fits the training data in N dimensional state space. The width of the nonlinear line is then characterized by the statistical characteristics of the training patterns. Stability of the recurrent network is verified by analyzing the trajectory of the points in the state space during convergence. The performance of the network is benchmarked through the reconstruction of original gray-scale images from their corrupted versions. It is observed that the proposed method can quickly and successfully reconstruct each image with an average convergence rate of 3.10 iterations.     

一种基于抗体网络的免疫算法

阐述了免疫系统抗体网络的机理和特点,深入分析了抗体网络与常用的免疫算法和Hopfield神经网络异同.通过不断更新输入模式(抗原)和采用最优保存策略,将基于克隆选择的竞争学习算子、自动生成网络结构、剪枝算子和低频变异用于进化操作,提出一种新的基于抗体网络的免疫算法,用于函数优化问题.实验结果表明新算法可行有效.与常用的免疫算法、Hopfield神经网络优化算法比较,新算法具有较好的全局搜索能力和较快收敛速度.     

基于SAA-SSA-BPNN的网络安全态势评估模型

针对目前网络安全态势评估模型准确性和收敛性有待提高的问题,提出一种基于SAA-SSA-BPNN的网络安全态势评估模型。该模型利用模拟退火算法（SAA）可以一定概率接受劣解并有大概率跳出局部极值达到全局最优解的特性来优化麻雀搜索算法,利用优化后的麻雀搜索算法（SSA）具有良好稳定性和收敛速度快且不易陷入局部最优的特点对BP神经网络（BPNN）进行改进,找到最佳适应度个体并获取最优权值和阈值,将其作为初始值赋给BP神经网络,将预处理后的指标数据输入改进后的BP神经网络模型对其进行训练,利用训练好的模型对网络系统所遭受威胁的程度进行评估。对比实验结果表明,该评估模型比其他基于改进BP神经网络的态势评估模型准确性更高,收敛速度更快。     

基于回归神经网络自适应快速BP算法

动态递归网络Elman网络结构简单，运算量少，适合于实时系统辨识。以Elman网络结构推导了在线学习算法。针对于传统BP算法会产生局部收敛和收敛速度慢等缺点，提出了一种改进的自适应BP算法，运用到回归神经网络，提高了在线学习的速度与收敛速度，仿真实验表明了此算法的有效性和快速性。     

一种基于BP神经网络的数字识别新方法

针对标准BP神经网络收敛速度慢、易陷入局部极小点的缺点,提出了一种新的BP神经网络改进算法。该算法通过变步长法和牛顿法来改进BP算法,加快了网络的收敛速度,且收敛速度快于其他的改进算法。在此基础上将BP神经网络应用于数字识别中,为其网络建立识别模型。利用仿真实验观察BP网络的泛化能力以及识别准确性,比较BP算法及其改进方案,提出改进方案中分别需要注意的地方。     

连续搅拌反应釜( CSTR)的神经网络动态建模

基于广义性能指标,提出一种神经网络学习算法－广义递推预报误差学习算法（ＧＲＰＥ）,该算法具有二阶收敛阶次。同时讨论了学习速率的选择问题,利用所提出方法对ＣＳＴＲ动态建模结果表明,基于ＧＲＰＥ训练的ＤＲＮＮ比基于ＢＰ训练的ＭＬＰ模型精度高,收敛速度快。     

基于状态延迟动态递归神经网络的机器人动态自适应跟踪辨识

对一种在Elman动态递归网络基础上发展而来的复合输入动态递归网络(CIDRNN)作 了改进,提出一种新的动态递归神经网络结构,称为状态延迟动态递归神经网络(State Delay Input Dynamical Recurrent Neural Network).具有这种新的拓扑结构和学习规则的动态递归网 络,不仅明确了各权值矩阵的意义,而且使权值的训练过程更为简洁,意义更为明确.仿真实验 表明,这种结构的网络由于增加了网络输入输出的前一步信息,提高了收敛速度,增强了实时 控制的可能性.然后将该网络用于机器人未知非线性动力学的辨识中,使用辨识实际输出与机理 模型输出之间的偏差,来识别机理模型或简化模型所丢失的信息,既利用了机器人现有的建模 方法,又可以减小网络运算量,提高辨识速度.仿真结果表明了这种改进的有效性.     

改进的自适应脊波网络的碳通量预测

碳通量同生态因素之间具有复杂的非线性关系,可以通过生态因素预测碳通量。为提高网络的训练速度和预测精度,针对碳通量数据高维、多样本、非线性、超平面奇异的特点,提出了一种改进的自适应脊波网络预测模型,采用高斯牛顿法调整激励函数的参数,运用矩阵分块法和伪逆矩阵计算脊波网络的权值和阈值。通过实验,比较了改进自适应脊波网络、自适应脊波网络和小波网络的训练收敛速度、隐含层节点个数和预测精度。实验结果表明,提出的预测模型预测精度更高,网络结构更稀疏,训练收敛速度更快。     

A simulation study of artificial neural networks for nonlinear time-series forecasting

This study presents an experimental evaluation of neural networks for nonlinear time-series forecasting. The effects of three main factors — input nodes, hidden nodes and sample size, are examined through a simulated computer experiment. Results show that neural networks are valuable tools for modeling and forecasting nonlinear time series while traditional linear methods are not as competent for this task. The number of input nodes is much more important than the number of hidden nodes in neural network model building for forecasting. Moreover, large sample is helpful to ease the overfitting problem.Scope and purposeInterest in using artificial neural networks for forecasting has led to a tremendous surge in research activities in the past decade. Yet, mixed results are often reported in the literature and the effect of key modeling factors on performance has not been thoroughly examined. The lack of systematic approaches to neural network model building is probably the primary cause of inconsistencies in reported findings. In this paper, we present a systematic investigation of the application of neural networks for nonlinear time-series analysis and forecasting. The purpose is to have a detailed examination of the effects of certain important neural network modeling factors on nonlinear time-series modeling and forecasting.     

基于数值优化的改进BP算法在旋转机械故障诊断中的应用

机械设备的安全运行对企业的现代化生产至关重要,因而对故障机械的诊断近年来受到了普遍关注,而神经网络具有分辨原因及故障类型的能力,在故障诊断领域中得到了广泛应用.本文针对传统BP算法存在的收敛速度慢以及容易陷入局部最小点等问题,给出了两种基于数值优化方法的改进BP算法,应用改进的BP算法对旋转机械故障进行诊断研究,结果表明,加快了网络的收敛速度.证明该算法比BP算法精度更高且收敛速度更快.