Stability analysis of discrete-time recurrent neural networks

We address the problem of global Lyapunov stability of discrete-time recurrent neural networks (RNNs) in the unforced (unperturbed) setting. It is assumed that network weights are fixed to some values, for example, those attained after training. Based on classical results of the theory of absolute stability, we propose a new approach for the stability analysis of RNNs with sector-type monotone nonlinearities and nonzero biases. We devise a simple state-space transformation to convert the original RNN equations to a form suitable for our stability analysis. We then present appropriate linear matrix inequalities (LMIs) to be solved to determine whether the system under study is globally exponentially stable. Unlike previous treatments, our approach readily permits one to account for non-zero biases usually present in RNNs for improved approximation capabilities. We show how recent results of others on the stability analysis of RNNs can be interpreted as special cases within our approach. We illustrate how to use our approach with examples. Though illustrated on the stability analysis of recurrent multilayer perceptrons, the approach proposed can also be applied to other forms of time-lagged RNNs.     

Developing Complex Systems Using Evolved Pattern Generators

Self-organization of connection patterns within brain areas of animals begins prenatally, and has been shown to depend on internally generated patterns of neural activity. The neural structures continue to develop postnatally through externally driven patterns, when the sensory systems are exposed to stimuli from the environment. The internally generated patterns have been proposed to give the neural system an appropriate bias so that it can learn reliably from complex environmental stimuli. This paper evaluates the hypothesis that complex artificial learning systems can benefit from a similar approach, consisting of initial training with patterns from an evolved pattern generator, followed by training with the actual training set. To test this hypothesis, competitive learning networks were trained for recognizing handwritten digits. The results demonstrate how the approach can improve learning performance by discovering the appropriate initial weight biases, thereby compensating for weaknesses of the learning algorithm. Due to the smaller evolutionary search space, this approach was also found to require much fewer generations than direct evolution of network weights. Since discovering the right biases efficiently is critical for solving large-scale problems with learning, these results suggest that internal training pattern generation is an effective method for constructing complex systems     

Training trajectories by continuous recurrent multilayer networks

This paper addresses the problem of training trajectories by means of continuous recurrent neural networks whose feedforward parts are multilayer perceptrons. Such networks can approximate a general nonlinear dynamic system with arbitrary accuracy. The learning process is transformed into an optimal control framework where the weights are the controls to be determined. A training algorithm based upon a variational formulation of Pontryagin's maximum principle is proposed for such networks. Computer examples demonstrating the efficiency of the given approach are also presented.     

Hopfield Neural Networks for Parametric Identification of Dynamical Systems

In this work, a novel method, based upon Hopfield neural networks, is proposed for parameter estimation, in the context of system identification. The equation of the neural estimator stems from the applicability of Hopfield networks to optimization problems, but the weights and the biases of the resulting network are time-varying, since the target function also varies with time. Hence the stability of the method cannot be taken for granted. In order to compare the novel technique and the classical gradient method, simulations have been carried out for a linearly parameterized system, and results show that the Hopfield network is more efficient than the gradient estimator, obtaining lower error and less oscillations. Thus the neural method is validated as an on-line estimator of the time-varying parameters appearing in the model of a nonlinear physical system.     

Generalized neuron-based PSS and adaptive PSS

Artificial neural networks can be used as intelligent controllers to control non-linear, dynamic systems through learning, which can easily accommodate the non-linearities and time dependencies. However, they require large training time and large number of neurons to deal with complex problems. Taking benefit of the characteristics of a Generalized Neuron that requires much smaller training data and shorter training time, a Generalized Neuron-Based Power System Stabilizer (GNPSS) and an adaptive version of the same have been developed. The objective of this paper is to compare the performance of the GNPSS with that of an adaptive version, the weights of which are updated on-line.     

非线性伺服电动机的神经网络逆控制

伺服电动机由于存在接触过程的非线性、温漂等非线性因素的影响,很难建立其精确的数学模型,使得基于数学模型的控制困难.针对伺服电动机存在的非线性问题,提出了一种新颖的基于BP神经网络直接逆控制方法.首先,利用BP神经网络建立系统的正向模型(NNI),然后,设计基于神经网络的直接逆控制器(NNC),实现了对伺服电动机的自适应控制.在Lyapunov稳定性分析的基础上,给出了BP算法学习算子的选择方案,保证神经网络权值训练的快速收敛,同时,对训练BP神经网络控制器的专用算法(specialized learning)进行改进,利用NNI的输出求取权值调整的灵敏度函数.数字仿真结果表明提出的控制算法是简单有效的.     

Topology optimization of neural networks based on a coupled genetic algorithm and particle swarm optimization techniques (c-GA–PSO-NN)

In this short paper, a coupled genetic algorithm and particle swarm optimization technique was used to supervise neural networks where the applied operators and connections of layers were tracked by genetic algorithm and numeric values of biases and weights of layers were examined by particle swarm optimization to modify the optimal network topology. The method was applied for a previously studied case, and results were analyzed. The convergence to the optimal topology was highly fast and efficient, and the obtained weights and biases revealed great reliability in reproduction of data. The optimal topology of neural networks was obtained only after seven iterations, and an average square of the correlation (R ²) of 0.9989 was obtained for the studied cases. The proposed method can be used for fast and reliable topology optimization of neural networks.

     

未知光源位置环境中物体形状恢复的神经网络方法研究

用神经网络方法解决未知光源位置环境中物体三维形状恢复的问题.对漫反射表面,用神经网络方法由已知表面形状物体及其对应图像的灰度值进行学习,所得权值可视为环境光源参数.由此可恢复同样光源环境中其它物体的三维形状.实验证明,神经网络方法可以解决未知光源位置环境(包括多个光源)中漫反射表面物体的三维形状恢复问题.     

A rapid learning and dynamic stepwise updating algorithm for flatneural networks and the application to time-series prediction

A fast learning algorithm is proposed to find an optimal weights of the flat neural networks (especially, the functional-link network). Although the flat networks are used for nonlinear function approximation, they can be formulated as linear systems. Thus, the weights of the networks can be solved easily using a linear least-square method. This formulation makes it easier to update the weights instantly for both a new added pattern and a new added enhancement node. A dynamic stepwise updating algorithm is proposed to update the weights of the system on-the-fly. The model is tested on several time-series data including an infrared laser data set, a chaotic time-series, a monthly flour price data set, and a nonlinear system identification problem. The simulation results are compared to existing models in which more complex architectures and more costly training are needed. The results indicate that the proposed model is very attractive to real-time processes.     

Intelligent stock trading system with price trend prediction and reversal recognition using dual-module neural networks

This article presents an intelligent stock trading system that can generate timely stock trading suggestions according to the prediction of short-term trends of price movement using dual-module neural networks(dual net). Retrospective technical indicators extracted from raw price and volume time series data gathered from the market are used as independent variables for neural modeling. Both neural network modules of thedual net learn the correlation between the trends of price movement and the retrospective technical indicators by use of a modified back-propagation learning algorithm. Reinforcing the temporary correlation between the neural weights and the training patterns, dual modules of neural networks are respectively trained on a short-term and a long-term moving-window of training patterns. An adaptive reversal recognition mechanism that can self-tune thresholds for identification of the timing for buying or selling stocks has also been developed in our system. It is shown that the proposeddual net architecture generalizes better than one single-module neural network. According to the features of acceptable rate of returns and consistent quality of trading suggestions shown in the performance evaluation, an intelligent stock trading system with price trend prediction and reversal recognition can be realized using the proposed dual-module neural networks.     

AUV深度的神经网络辨识和学习控制仿真研究

将自主水下航行器（AUV）的深度控制问题转换为对非线性严格反馈系统的分析,提出了一种结合反步法和确定学习理论的自适应学习控制方法。通过反步法设计了一种输入状态稳定（ISS）神经网络控制器,其中引入小增益定理,避免了控制器设计中存在的奇异值问题,并在满足持续激励（PE）条件下,利用神经网络辨识实现了对系统未知动态的局部准确逼近和部分神经网络权值的收敛,保证了闭环系统的稳定。将从动态模式中学到的知识静态保存,提取动态特征设计学习控制器,仿真结果表明,该控制器避免了执行同样任务时的重复训练,改善了系统控制性能,验证了所提控制方法的有效性。     

A New Adaptive Backpropagation Algorithm Based on Lyapunov Stability Theory for Neural Networks

A new adaptive backpropagation (BP) algorithm based on Lyapunov stability theory for neural networks is developed in this paper. It is shown that the candidate of a Lyapunov function V(k) of the tracking error between the output of a neural network and the desired reference signal is chosen first, and the weights of the neural network are then updated, from the output layer to the input layer, in the sense that DeltaV(k)=V(k)-V(k-1)<0. The output tracking error can then asymptotically converge to zero according to Lyapunov stability theory. Unlike gradient-based BP training algorithms, the new Lyapunov adaptive BP algorithm in this paper is not used for searching the global minimum point along the cost-function surface in the weight space, but it is aimed at constructing an energy surface with a single global minimum point through the adaptive adjustment of the weights as the time goes to infinity. Although a neural network may have bounded input disturbances, the effects of the disturbances can be eliminated, and asymptotic error convergence can be obtained. The new Lyapunov adaptive BP algorithm is then applied to the design of an adaptive filter in the simulation example to show the fast error convergence and strong robustness with respect to large bounded input disturbances     

Wavelet networks for nonlinear system modeling

This study presents a nonlinear systems and function learning by using wavelet network. Wavelet networks are as neural network for training and structural approach. But, training algorithms of wavelet networks is required a smaller number of iterations when the compared with neural networks. Gaussian-based mother wavelet function is used as an activation function. Wavelet networks have three main parameters; dilation, translation, and connection parameters (weights). Initial values of these parameters are randomly selected. They are optimized during training (learning) phase. Because of random selection of all initial values, it may not be suitable for process modeling. Because wavelet functions are rapidly vanishing functions. For this reason heuristic procedure has been used. In this study serial-parallel identification model has been applied to system modeling. This structure does not utilize feedback. Real system outputs have been exercised for prediction of the future system outputs. So that stability and approximation of the network is guaranteed. Gradient methods have been applied for parameters updating with momentum term. Quadratic cost function is used for error minimization. Three example problems have been examined in the simulation. They are static nonlinear functions and discrete dynamic nonlinear system.     

集成重复训练极限学习机的数据分类

极限学习机是一种随机化算法,它随机生成单隐含层神经网络输入层连接权和隐含层偏置,用分析的方法确定输出层连接权。给定网络结构,用极限学习机重复训练网络,会得到不同的学习模型。本文提出了一种集成模型对数据进行分类的方法。首先用极限学习机算法重复训练若干个单隐含层前馈神经网络,然后用多数投票法集成训练好的神经网络,最后用集成模型对数据进行分类,并在10个数据集上和极限学习机及集成极限学习机进行了实验比较。实验结果表明,本文提出的方法优于极限学习机和集成极限学习机。     

一种k均值和神经网络集成的语音识别方法

提出了一种基于k均值聚类和BP神经网络集成的语音识别方法,该方法以神经网络集成模型为基础,利用k均值聚类算法选择部分有差异性的个体神经网络再进行集成学习,既克服了单个BP网络模型容易局部收敛和不稳定性的缺点,又解决了传统集成方法训练时间长和个体网络差异性不明显的问题。通过对非特定人孤立词的语音识别的实验,证实了该方法的有效性。     

Real time adaptive nonlinear model inversion control of a twin rotor MIMO system using neural networks

This paper investigates the development and experimental implementation of an adaptive dynamic nonlinear model inversion control law for a Twin Rotor MIMO System (TRMS) using artificial neural networks. The TRMS is a highly nonlinear aerodynamic test rig with complex cross-coupled dynamics and therefore represents the control challenges of modern air vehicles. A highly nonlinear 1DOF mathematical model of the TRMS is considered in this study and a nonlinear inverse model is developed for the pitch channel of the system. An adaptive neural network element is integrated thereafter with the feedback control system to compensate for model inversion errors. The proposed on-line learning algorithm updates the weights and biases of the neural network using the error between the set-point and the real output. The real-time response of the method shows a satisfactory tracking performance in the presence of inversion errors caused by model uncertainty. The approach is therefore deemed to be suitable to apply real-time to other nonlinear systems with necessary modifications.     

基于遗传策略和神经网络的非监督分类方法

文章提出了一种新的基于遗传策略和模糊ART(adaptive resonance theory)神经网络的非监督分类方法.首先,利用原有的训练样本对模糊ART神经网络进行非监督训练,然后,采用遗传策略为模糊ART神经网络增加各类族边界邻域内的训练样本点,再对模糊ART神经网络进行有监督训练.这种方法解决了训练样本在较少条件下的ART系列神经网络的学习与分类问题,提高了ART系列神经网络的分类性能,并扩展了其应用范围.     

Feedback-Linearization-Based Neural Adaptive Control for Unknown Nonaffine Nonlinear Discrete-Time Systems

A new feedback-linearization-based neural network (NN) adaptive control is proposed for unknown nonaffine nonlinear discrete-time systems. An equivalent model in affine-like form is first derived for the original nonaffine discrete-time systems as feedback linearization methods cannot be implemented for such systems. Then, feedback linearization adaptive control is implemented based on the affine-like equivalent model identified with neural networks. Pretraining is not required and the weights of the neural networks used in adaptive control are directly updated online based on the input–output measurement. The dead-zone technique is used to remove the requirement of persistence excitation during the adaptation. With the proposed neural network adaptive control, stability and performance of the closed-loop system are rigorously established. Illustrated examples are provided to validate the theoretical findings.      

Parallel evolutionary training algorithms for “hardware-friendly” neural networks

In this paper, Parallel Evolutionary Algorithms for integer weightneural network training are presented. To this end, each processoris assigned a subpopulation of potential solutions. Thesubpopulations are independently evolved in parallel andoccasional migration is employed to allow cooperation betweenthem. The proposed algorithms are applied to train neural networksusing threshold activation functions and weight values confined toa narrow band of integers. We constrain the weights and biases inthe range [–3, 3], thus they can be represented by just 3 bits.Such neural networks are better suited for hardware implementationthan the real weight ones. These algorithms have been designedkeeping in mind that the resulting integer weights require lessbits to be stored and the digital arithmetic operations betweenthem are easier to be implemented in hardware. Another advantageof the proposed evolutionary strategies is that they are capableof continuing the training process ``on-chip', if needed. Ourintention is to present results of parallel evolutionaryalgorithms on this difficult task. Based on the application of theproposed class of methods on classical neural network problems,our experience is that these methods are effective and reliable.     

Multiple neural-network-based adaptive controller using orthonormalactivation function neural networks

A direct adaptive control scheme is developed using orthonormal activation function-based neural networks (OAFNNs) for trajectory tracking control of a class of nonlinear systems. Multiple OAFNNs are employed in these controllers for feedforward compensation of unknown system dynamics. Choice of multiple OAFNNs allows a reduction in overall network size reducing the computational requirements. The network weights are tuned online, in real time. The overall stability of the system and the neural networks is guaranteed using Lyapunov analysis. The developed neural controllers are evaluated experimentally and the experimental results are shown to support theoretical analysis. The effects of network parameters on system performance are experimentally evaluated and are presented. The superior learning capability of OAFNNs is demonstrated through experimental results. The OAFNNs were able to model the true nature of the nonlinear system dynamics characteristics for a rolling-sliding contact as well as for stiction.