Differential Evolution and Levenberg Marquardt Trained Neural Network Scheme for Nonlinear System Identification

This paper proposes a new nonlinear system identification scheme using differential evolution (DE), neural network and Levenberg Marquardt algorithm (LM). Here, DE and LM in a combined framework are used to train a neural network for achieving better convergence of neural network weight optimization. A number of examples including a practical case-study have been considered for implementation of different system identification methods namely, only NN, DE+NN and DE+LM+NN. After, a series of simulation studies of these methods on the different nonlinear systems it has been confirmed that the proposed DE and LM trained NN approach to nonlinear system identification has yielded better identification results in terms of time of convergence and less identification error.     

Nonlinear system identification using memetic differential evolution trained neural networks

Several gradient-based approaches such as back propagation (BP) and Levenberg Marquardt (LM) methods have been developed for training the neural network (NN) based systems. But, for multimodal cost functions these procedures may lead to local minima, therefore, the evolutionary algorithms (EAs) based procedures are considered as promising alternatives. In this paper we focus on a memetic algorithm based approach for training the multilayer perceptron NN applied to nonlinear system identification. The proposed memetic algorithm is an alternative to gradient search methods, such as back-propagation and back-propagation with momentum which has inherent limitations of many local optima. Here we have proposed the identification of a nonlinear system using memetic differential evolution (DE) algorithm and compared the results with other six algorithms such as Back-propagation (BP), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Differential Evolution (DE), Genetic Algorithm Back-propagation (GABP), Particle Swarm Optimization combined with Back-propagation (PSOBP). In the proposed system identification scheme, we have exploited DE to be hybridized with the back propagation algorithm, i.e. differential evolution back-propagation (DEBP) where the local search BP algorithm is used as an operator to DE. These algorithms have been tested on a standard benchmark problem for nonlinear system identification to prove their efficacy. First examples shows the comparison of different algorithms which proves that the proposed DEBP is having better identification capability in comparison to other. In example 2 good behavior of the identification method is tested on an one degree of freedom (1DOF) experimental aerodynamic test rig, a twin rotor multi-input-multi-output system (TRMS), finally it is applied to Box and Jenkins Gas furnace benchmark identification problem and its efficacy has been tested through correlation analysis.     

An improved SPSA algorithm for system identification using fuzzy rules for training neural networks

Simultaneous perturbation stochastic approximation (SPSA) belongs to the class of gradient-free optimization methods that extract gradient information from successive objective function evaluation. This paper describes an improved SPSA algorithm, which entails fuzzy adaptive gain sequences, gradient smoothing, and a step rejection procedure to enhance convergence and stability. The proposed fuzzy adaptive simultaneous perturbation approximation (FASPA) algorithm is particularly well suited to problems involving a large number of parameters such as those encountered in nonlinear system identification using neural networks (NNs). Accordingly, a multilayer perceptron (MLP) network with popular training algorithms was used to predicate the system response. We found that an MLP trained by FASPSA had the desired accuracy that was comparable to results obtained by traditional system identification algorithms. Simulation results for typical nonlinear systems demonstrate that the proposed NN architecture trained with FASPSA yields improved system identification as measured by reduced time of convergence and a smaller identification error.     

A differential evolution based neural network approach to nonlinear system identification

This paper addresses the effectiveness of soft computing approaches such as evolutionary computation (EC) and neural network (NN) to system identification of nonlinear systems. In this work, two evolutionary computing approaches namely differential evolution (DE) and opposition based differential evolution (ODE) combined with Levenberg Marquardt algorithm have been considered for training the feed-forward neural network applied for nonlinear system identification. Results obtained envisage that the proposed combined opposition based differential evolution neural network (ODE-NN) approach to identification of nonlinear system exhibits better model identification accuracy compared to differential evolution neural network (DE-NN) approach. The above method is finally tested on a one degree of freedom (1DOF) highly nonlinear twin rotor multi-input–multi-output system (TRMS) to verify the identification performance.     

Nonlinear systems identification using dynamic multi-time scale neural networks

In this paper, two Neural Network (NN) identifiers are proposed for nonlinear systems identification via dynamic neural networks with different time scales including both fast and slow phenomena. The first NN identifier uses the output signals from the actual system for the system identification. The on-line update laws for dynamic neural networks have been developed using the Lyapunov function and singularly perturbed techniques. In the second NN identifier, all the output signals from nonlinear system are replaced with the state variables of the neuron networks. The on-line identification algorithm with dead-zone function is proposed to improve nonlinear system identification performance. Compared with other dynamic neural network identification methods, the proposed identification methods exhibit improved identification performance. Three examples are given to demonstrate the effectiveness of the theoretical results.     

Modeling of a proton exchange membrane fuel cell based on the hybrid particle swarm optimization with Levenberg–Marquardt neural network

This paper presents a nonlinear modeling approach of a proton exchange membrane fuel cell (PEMFC) based on the hybrid particle swarm optimization with Levenberg–Marquardt algorithm neural network (PSO-LM NN). The PSO algorithm converges rapidly during the initial stages of a global search, while it becomes extremely slow around the global optimum. On the contrary, the LM algorithm can achieve faster convergent speed around the global optimum, while it is prone to being trapped in the local minimum. Therefore the hybrid algorithm with a transition from PSO search to LM training is proposed to train the weights and thresholds of neural network, which aims to exploit the advantage of the both algorithms. An accurate mathematical model is an extremely useful tool for the fuel cell design, and neural network is an excellent optional tool for complex nonlinear dynamic system modeling such as PEMFC. In the paper, firstly a highly reduced PEMFC dynamic physical model is established to generate the data for the PSO-LM NN model training and validation, and then the neural network nonlinear autoregressive model based on the PSO-LM algorithm is applied in modeling PEMFC voltage and temperature model, and finally the validation test result demonstrates that the trained PSO-LM NN model can efficiently approach the dynamic behavior of a PEMFC.     

Online adaptive optimal control for continuous-time nonlinear systems with completely unknown dynamics

An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.     

Fast and improved backpropagation learning of multi-layer artificial neural network using adaptive activation function

In the conventional backpropagation (BP) learning algorithm used for the training of the connecting weights of the artificial neural network (ANN), a fixed slope−based sigmoidal activation function is used. This limitation leads to slower training of the network because only the weights of different layers are adjusted using the conventional BP algorithm. To accelerate the rate of convergence during the training phase of the ANN, in addition to updates of weights, the slope of the sigmoid function associated with artificial neuron can also be adjusted by using a newly developed learning rule. To achieve this objective, in this paper, new BP learning rules for slope adjustment of the activation function associated with the neurons have been derived. The combined rules both for connecting weights and slopes of sigmoid functions are then applied to the ANN structure to achieve faster training. In addition, two benchmark problems: classification and nonlinear system identification are solved using the trained ANN. The results of simulation-based experiments demonstrate that, in general, the proposed new BP learning rules for slope and weight adjustments of ANN provide superior convergence performance during the training phase as well as improved performance in terms of root mean square error and mean absolute deviation for classification and nonlinear system identification problems.     

Improvement of the neighborhood based Levenberg-Marquardt algorithm by local adaptation of the learning coefficient

In this letter, an improvement of the recently developed neighborhood-based Levenberg-Marquardt (NBLM) algorithm is proposed and tested for neural network (NN) training. The algorithm is modified by allowing local adaptation of a different learning coefficient for each neighborhood. This simple add-in to the NBLM training method significantly increases the efficiency of the training episodes carried out with small neighborhood sizes, thus, allowing important savings in memory occupation and computational time while obtaining better performance than the original Levenberg-Marquardt (LM) and NBLM methods.     

Neural-Network-Based Control for Discrete-Time Nonlinear Systems with Input Saturation Under Stochastic Communication Protocol

In this paper,an adaptive dynamic programming(ADP)strategy is investigated for discrete-time nonlinear systems with unknown nonlinear dynamics subject to input saturation.To save the communication resources between the controller and the actuators,stochastic communication protocols(SCPs)are adopted to schedule the control signal,and therefore the closed-loop system is essentially a protocol-induced switching system.A neural network(NN)-based identifier with a robust term is exploited for approximating the unknown nonlinear system,and a set of switch-based updating rules with an additional tunable parameter of NN weights are developed with the help of the gradient descent.By virtue of a novel Lyapunov function,a sufficient condition is proposed to achieve the stability of both system identification errors and the update dynamics of NN weights.Then,a value iterative ADP algorithm in an offline way is proposed to solve the optimal control of protocol-induced switching systems with saturation constraints,and the convergence is profoundly discussed in light of mathematical induction.Furthermore,an actor-critic NN scheme is developed to approximate the control law and the proposed performance index function in the framework of ADP,and the stability of the closed-loop system is analyzed in view of the Lyapunov theory.Finally,the numerical simulation results are presented to demonstrate the effectiveness of the proposed control scheme.     

Design of optimal PID controller using hybrid differential evolution and particle swarm optimization with an aging leader and challengers

This paper presents a new algorithm designed to find the optimal parameters of PID controller. The proposed algorithm is based on hybridizing between differential evolution (DE) and Particle Swarm Optimization with an aging leader and challengers (ALC-PSO) algorithms. The proposed algorithm (ALC-PSODE) is tested on twelve benchmark functions to confirm its performance. It is found that it can get better solution quality, higher success rate in finding the solution and yields in avoiding unstable convergence. Also, ALC-PSODE is used to tune PID controller in three tanks liquid level system which is a typical nonlinear control system. Compared to different PSO variants, genetic algorithm (GA), differential evolution (DE) and Ziegler–Nichols method; the proposed algorithm achieve the best results with least standard deviation for different swarm size. These results show that ALC-PSODE is more robust and efficient while keeping fast convergence.     

基于混沌策略状态转移算法的混沌系统参数辨识*

针对混沌系统的参数辨识是一个多维参数的优化问题,提出了基于混沌策略状态转移算法的混沌系统参数辨识方法。该方法是在初始化时以混沌序列初始化种群,在搜索过程中引入混沌变异机制,利用遍历性对状态进行变异操作,避免了过早收敛,提高了全局搜索能力。利用该算法辨识Lorenz混沌系统参数,并与基本状态转移算法和粒子群算法进行比较。仿真结果表明,在有无噪声干扰的情况下,该算法比粒子群算法和基本状态转移算法具有更好的辨识精度且比粒子群算法具有更好的收敛速度,证明了该算法的有效性和抗干扰性,对混沌理论的发展有重要的意义。     

A hybridization of teaching–learning-based optimization and differential evolution for chaotic time series prediction

Chaotic time series prediction problems have some very interesting properties and their prediction has received increasing interest in the recent years. Prediction of chaotic time series based on the phase space reconstruction theory has been applied in many research fields. It is well known that prediction of a chaotic system is a nonlinear, multivariable and multimodal optimization problem for which global optimization techniques are required in order to avoid local optima. In this paper, a new hybrid algorithm named teaching–learning-based optimization (TLBO)–differential evolution (DE), which integrates TLBO and DE, is proposed to solve chaotic time series prediction. DE is incorporated into update the previous best positions of individuals to force TLBO jump out of stagnation, because of its strong searching ability. The proposed hybrid algorithm speeds up the convergence and improves the algorithm’s performance. To demonstrate the effectiveness of our approaches, ten benchmark functions and three typical chaotic nonlinear time series prediction problems are used for simulating. Conducted experiments indicate that the TLBO–DE performs significantly better than, or at least comparable to, TLBO and some other algorithms.     

基于Levenberg-Marquardt算法和最小二乘方法的小波网络混合学习算法

本文针对小波网络现有学习算法的不足,把Levenberg-Marquardt算法(简称LM算法)和最小二乘算法有机地结合在一起,提出了一种新的小波网络混合学习算法.在该混合算法中LM算法用来训练小波网络的非线性参数,而最小二乘算法用来训练线性参数.最后以辩识一个混沌系统为例进行了数值仿真,并与改进的BP算法和单纯LM算法进行了比较,结果说明了所提算法具有很好的收敛性能和收敛速度.     

Optimal short-term hydro-thermal scheduling using quasi-oppositional teaching learning based optimization

This paper presents a new approach for solving short-term hydrothermal scheduling (HTS) using an integrated algorithm based on teaching learning based optimization (TLBO) and oppositional based learning (OBL). The practical hydrothermal system is highly complex and possesses nonlinear relationship of the problem variables, cascading nature of hydro reservoirs, water transport delay and scheduling time linkage that make the problem of optimization difficult using standard optimization methods. To overcome these problems, the proposed quasi-oppositional teaching learning based optimization (QOTLBO) is employed. To show its efficiency and robustness, the proposed QOTLBO algorithm is applied on two test systems. Numerical results of QOTLBO are compared with those obtained by two phase neural network, augmented Lagrange method, particle swarm optimization (PSO), improved self-adaptive PSO (ISAPSO), improved PSO (IPSO), differential evolution (DE), modified DE (MDE), fuzzy based evolutionary programming (Fuzzy EP), clonal selection algorithm (CSA) and TLBO approaches. The simulation results reveal that the proposed algorithm appears to be the best in terms of convergence speed, solution time and minimum cost when compared with other established methods. This method is considered to be a promising alternative approach for solving the short-term HTS problems in practical power system.     

双群体伪并行差分进化算法研究及应用

为了提高差分进化算法的全局搜索能力和收敛速率,本文提出了一种双群体伪并行差分进化算法.该算法结合差分进化算法DE/best/2/bin变异方式局部搜索能力强、收敛速度快,和DE/rand/1/bin变异方式全局搜索能力强、鲁棒性好的特点,采用串行算法结构实现并行差分进化算法独立进化、信息交换的思想.为使初始化个体均匀分布在搜索空间,提高算法收敛到全局最优解的鲁棒性,提出了一种基于平均熵的初始化策略.典型Benchmarks函数测试和非线性系统模型参数估计结果表明,该方法能显著提高算法的收敛速率和全局搜索能力.     

A novel optimal PID controller autotuning design based on the SLP algorithm

A novel optimal proportional integral derivative (PID) autotuning controller design based on a new algorithm approach, the “swarm learning process” (SLP) algorithm, is proposed. It improves the convergence and performance of the autotuning PID parameter by applying the swarm and learning algorithm concepts. Its convergence is verified by two methods, global convergence and characteristic convergence. In the case of global convergence, the convergence rule of a random search algorithm is employed to judge, and Markov chain modelling is used to analyse. The superiority of the proposed method, in terms of characteristic convergence and performance, is verified through the simulation based on the automatic voltage regulator and direct current motor control system. Verification is performed by comparing the results of the proposed model with those of other algorithms, that is, the ant colony optimization with a new constrained Nelder–Mead algorithm, the genetic algorithm (GA), the particle swarm optimization (PSO) algorithm, and a neural network (NN). According to the global convergence analysis, the proposed method satisfies the convergence rule of the random search algorithm. With respect to the characteristic convergence and performance, the proposed method provides a better response than the GA, the PSO, and the NN for both control systems.     

基于单形正交实验设计的差分演化算法

为了克服传统差分演化(DE)算法在求解约束优化问题时出现的收敛性慢和容易陷入早熟等缺陷,提出一种新的基于单形正交实验设计的差分演化(SO-DE)算法。该算法设计了一种结合单形交叉和正交实验设计的混合交叉算子来提高差分演化算法的搜索能力;同时采用了一种改进的个体优劣比较准则对种群个体进行比较和选择。这种新的混合交叉算子利用多个父代个体进行单形交叉产生多个子代个体,从两者中选择优秀个体进行正交实验设计得到下一代种群个体。改进的个体优劣比较准则对不同状态下的种群采用不同的处理方案,其目的在于能够有效地权衡目标函数值和约束违反量之间的关系,从而选择优秀个体进入下一代种群。通过对13个标准测试函数和2个工程设计问题进行仿真实验,实验结果表明SO-DE算法求解的精度和标准方差都要优于HEAA算法和COEA/OED算法。SO-DE算法具有更高的精度以及更好的稳定性。     

A prediction-error-method for recursive identification of nonlinear systems

Nonlinear systems can be identified with nonlinear filters by combined estimation of parameters and state. Often these methods are very complex or have convergence problems, as for example the Extended Kalman Filter. In this paper a new algorithm for recursive nonlinear system identification is presented. Convergence problems are eliminated by an improved calculation of the gradient as the total derivative of the prediction error. By separately estimating parameters and states, computation time is reduced. The efficacy of the new identification algorithm is illustrated by studying its performance in a gravimetric filling system.     

基于自适应扩展卡尔曼滤波与神经网络的HPA预失真算法

针对强记忆功放的非线性问题,提出一种基于自适应扩展卡尔曼滤波与神经网络的高功放(High power amplifier, HPA)预失真算法.采用实数固定延时神经网络(Real-valued focused time-delay neural network, RVFTDNN)对间接学习结构预失真系统中的预失真器和逆估计器进行建模,扩展卡尔曼滤波(Extended Kalman filter, EKF)算法训练神经网络,从理论上指出Levenberg-Marquardt(LM)算法是EKF算法的特殊情况,并用李亚普诺夫稳定性理论分析EKF算法的稳定收敛条件,推导出测量误差矩阵的自适应迭代公式.结果表明:自适应EKF算法的训练误差和泛化误差均比LM算法更低,预失真后的邻道功率比(Adjacent channel power ratio, ACPR)比LM算法改善了2dB.