Stability analysis of discrete-time recurrent neural networks

We address the problem of global Lyapunov stability of discrete-time recurrent neural networks (RNNs) in the unforced (unperturbed) setting. It is assumed that network weights are fixed to some values, for example, those attained after training. Based on classical results of the theory of absolute stability, we propose a new approach for the stability analysis of RNNs with sector-type monotone nonlinearities and nonzero biases. We devise a simple state-space transformation to convert the original RNN equations to a form suitable for our stability analysis. We then present appropriate linear matrix inequalities (LMIs) to be solved to determine whether the system under study is globally exponentially stable. Unlike previous treatments, our approach readily permits one to account for non-zero biases usually present in RNNs for improved approximation capabilities. We show how recent results of others on the stability analysis of RNNs can be interpreted as special cases within our approach. We illustrate how to use our approach with examples. Though illustrated on the stability analysis of recurrent multilayer perceptrons, the approach proposed can also be applied to other forms of time-lagged RNNs.     

Continuous attractors of discrete-time recurrent neural networks

This paper studies the continuous attractors of discrete-time recurrent neural networks. Networks in discrete time can directly provide algorithms for efficient implementation in digital hardware. Continuous attractors of neural networks have been used to store and manipulate continuous stimuli for animals. A continuous attractor is defined as a connected set of stable equilibrium points. It forms a lower dimensional manifold in the original state space. Under some conditions, the complete analytical expressions for the continuous attractors of discrete-time linear recurrent neural networks as well as discrete-time linear-threshold recurrent neural networks are derived. Examples are employed to illustrate the theory.     

Adaptive feedback linearization control of chaotic systems via recurrent high-order neural networks

In the realm of nonlinear control, feedback linearization via differential geometric techniques has been a concept of paramount importance. However, the applicability of this approach is quite limited, in the sense that a detailed knowledge of the system nonlinearities is required. In practice, most physical chaotic systems have inherent unknown nonlinearities, making real-time control of such chaotic systems still a very challenging area of research. In this paper, we propose using the recurrent high-order neural network for both identifying and controlling unknown chaotic systems, in which the feedback linearization technique is used in an adaptive manner. The global uniform boundedness of parameter estimation errors and the asymptotic stability of tracking errors are proved by the Lyapunov stability theory and the LaSalle-Yoshizawa theorem. In a systematic way, this method enables stabilization of chaotic motion to either a steady state or a desired trajectory. The effectiveness of the proposed adaptive control method is illustrated with computer simulations of a complex chaotic system.     

Approximation of discrete-time state-space trajectories usingdynamic recurrent neural networks

In this note, the approximation capability of a class of discrete-time dynamic recurrent neural networks (DRNN's) is studied. Analytical results presented show that some of the states of such a DRNN described by a set of difference equations may be used to approximate uniformly a state-space trajectory produced by either a discrete-time nonlinear system or a continuous function on a closed discrete-time interval. This approximation process, however, has to be carried out by an adaptive learning process. This capability provides the potential for applications such as identification and adaptive control     

Bounds of the incremental gain for discrete-time recurrent neural networks

As a nonlinear system, a recurrent neural network generally has an incremental gain different from its induced norm. While most of the previous research efforts were focused on the latter, this paper presents a method to compute an effective upper bound of the former for a class of discrete-time recurrent neural networks, which is not only applied to systems with arbitrary inputs but also extended to systems with small-norm inputs. The upper bound is computed by simple optimizations subject to linear matrix inequalities (LMIs). To demonstrate the wide connections of our results to problems in control, the servomechanism is then studied, where a feedforward neural network is designed to control the output of a recurrent neural network to track a set of trajectories. This problem can be converted into the synthesis of feedforward-feedback gains such that the incremental gain of a certain system is minimized. An algorithm to perform such a synthesis is proposed and illustrated with a numerical example.     

Stability analysis of discrete-time recurrent neural networks based on standard neural network models

In order to conveniently analyze the stability of various discrete-time recurrent neural networks (RNNs), including bidirectional associative memory, Hopfield, cellular neural network, Cohen-Grossberg neural network, and recurrent multiplayer perceptrons, etc., the novel neural network model, named standard neural network model (SNNM) is advanced to describe this class of discrete-time RNNs. The SNNM is the interconnection of a linear dynamic system and a bounded static nonlinear operator. By combining Lyapunov functional with S-Procedure, some useful criteria of global asymptotic stability for the discrete-time SNNMs are derived, whose conditions are formulated as linear matrix inequalities. Most delayed (or non-delayed) RNNs can be transformed into the SNNMs to be stability analyzed in a unified way. Some application examples of the SNNMs to the stability analysis of the discrete-time RNNs shows that the SNNMs make the stability conditions of the RNNs easily verified.     

Exponential Lagrange stability for impulses in discrete-time delayed recurrent neural networks

This paper focuses on the problem of exponential stability in the sense of Lagrange for impulses in discrete-time delayed recurrent neural networks. By establishing a delayed impulsive discrete inequality and a novel difference inequality, combining with inequality techniques, some novel sufficient conditions are obtained to ensure exponential Lagrange stability for impulses in discrete-time delayed recurrent neural networks. Meanwhile, exponentially convergent scope of neural network is given. Finally, several numerical simulations are given to demonstrate the effectiveness of our results.     

Acoustic-to-phonetic mapping using recurrent neural networks

This paper describes the application of artificial neural networks to acoustic-to-phonetic mapping. The experiments described are typical of problems in speech recognition in which the temporal nature of the input sequence is critical. The specific task considered is that of mapping formant contours to the corresponding CVC' syllable. We performed experiments on formant data extracted from the acoustic speech signal spoken at two different tempos (slow and normal) using networks based on the Elman simple recurrent network model. Our results show that the Elman networks used in these experiments were successful in performing the acoustic-to-phonetic mapping from formant contours. Consequently, we demonstrate that relatively simple networks, readily trained using standard backpropagation techniques, are capable of initial and final consonant discrimination and vowel identification for variable speech rates     

A critique of neural networks for discrete-time linear control

This paper discusses the use of multi-layer perceptron networks for linear or linearizable, adaptive feedback control schemes in a discrete-time environment. A close look is taken at the model structure selected and the extent of the resulting parametrization. A comparison is made with standard, non-perceptron algorithms, e.g. self-tuning control, and it is shown how gross over-parametrization can occur in the neural network case. Because of the resultant heavy computational burden and poor controller convergence, a strong case is made against the use of neural networks for discrete-time linear control.     

Generalized predictive control using recurrent fuzzy neural networks for industrial processes

This paper presents a design methodology for predictive control of industrial processes via recurrent fuzzy neural networks (RFNNs). A discrete-time mathematical model using RFNN is constructed and a learning algorithm adopting a recursive least squares (RLS) approach is employed to identify the unknown parameters in the model. A generalized predictive control (GPC) law with integral action is derived based on the minimization of a modified predictive performance criterion. The stability and steady-state performance of the resulting control system are studied as well. Two examples including the control of a nonlinear process and the control of a physical variable-frequency oil-cooling machine are used to demonstrate the effectiveness of the proposed method. Both results from numerical simulations and experiments show that the proposed method is capable of controlling industrial processes with satisfactory performance under setpoint and load changes.     

Multistability of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions

This paper studies the multistability of a class of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions. It addresses the nondivergence, global attractivity, and complete stability of the networks. Using the local inhibition, conditions for nondivergence are derived, which not only guarantee nondivergence, but also allow for the existence of multiequilibrium points. Under these nondivergence conditions, global attractive compact sets are obtained. Complete stability is studied via constructing novel energy functions and using the well-known Cauchy Convergence Principle. Examples and simulation results are used to illustrate the theory.     

New delay-dependent stability results for discrete-time recurrent neural networks with time-varying delay

This paper studies the problem of stability analysis for discrete-time recurrent neural networks (DRNNs) with time-varying delays. By using the discrete Jensen inequality and the sector bound conditions, a new less conservative delay-dependent stability criterion is established in terms of linear matrix inequalities (LMIs) under a weak assumption on the activation functions. By using a delay decomposition method, a further improved stability criterion is also derived. It is shown that the newly obtained results are less conservative than the existing ones. Meanwhile, the computational complexity of the newly obtained stability conditions is reduced since less variables are involved. A numerical example is given to illustrate the effectiveness and the benefits of the proposed method.     

Adaptive control of a nonlinear dc motor drive using recurrent neural networks

A model-following adaptive control structure is proposed for the speed control of a nonlinear motor drive system and the compensation of the nonlinearities. A recurrent artificial neural network is used for the online modeling and control of the nonlinear motor drive system with high static and Coulomb friction. The neural network is first trained off-line to learn the inverse dynamics of the motor drive system using a modified form of the decoupled extended Kalman filter algorithm. It is shown that the recurrent neural network structure combined with the inverse model control approach allows an effective direct adaptive control of the motor drive system. The performance of this method is validated experimentally on a dc motor drive system using a standard personal computer. The results obtained confirm the excellent disturbance rejection and tracking performance properties of the system.     

Real time identification and control of dynamic systems using recurrent neural networks

Recently, several recurrent neural networks for solving constraint optimization problems were developed. In this paper, we propose a novel approach to the use of a projection neural network for solving real time identification and control of time varying systems. In addition to low complexity and simple structure, the proposed neural network can solve wider classes of time varying systems compare with other neural networks that are used for optimization such as Hopfield neural networks. Simulation results demonstrate the effectiveness and characteristics of the proposed neural network compared with a Hopfield neural network.     

Robust dynamic surface control of flexible joint robots using recurrent neural networks

A robust neuro-adaptive controller for uncertain flexible joint robots is presented. This control scheme integrates H-infinity disturbance attenuation design and recurrent neural network adaptive control technique into the dynamic surface control framework. Two recurrent neural networks are used to adaptively learn the uncertain functions in a flexible joint robot. Then, the effects of approximation error and filter error on the tracking performance are attenuated to a prescribed level by the embedded H-infinity controller, so that the desired H-infinity tracking performance can be achieved. Finally, simulation results verify the effectiveness of the proposed control scheme.     

Financial volatility trading using recurrent neural networks

We simulate daily trading of straddles on financial indexes. The straddles are traded based on predictions of daily volatility differences in the indexes. The main predictive models studied are recurrent neural nets (RNN). Such applications have often been studied in isolation. However, due to the special character of daily financial time-series, it is difficult to make full use of RNN representational power. Recurrent networks either tend to overestimate noisy data, or behave like finite-memory sources with shallow memory; they hardly beat classical fixed-order Markov models. To overcome data nonstationarity, we use a special technique that combines sophisticated models fitted on a larger data set, with a fixed set of simple-minded symbolic predictors using only recent inputs. Finally, we compare our predictors with the GARCH family of econometric models designed to capture time-dependent volatility structure in financial returns. GARCH models have been used to trade volatility. Experimental results show that while GARCH models cannot generate any significantly positive profit, by careful use of recurrent networks or Markov models, the market makers can generate a statistically significant excess profit, but then there is no reason to prefer RNN over much more simple and straightforward Markov models. We argue that any report containing RNN results on financial tasks should be accompanied by results achieved by simple finite-memory sources combined with simple techniques to fight nonstationarity in the data.     

Two recurrent neural networks for local joint torque optimizationof kinematically redundant manipulators

This paper presents two neural network approaches to real-time joint torque optimization for kinematically redundant manipulators. Two recurrent neural networks are proposed for determining the minimum driving joint torques of redundant manipulators for the eases without and with taking the joint torque limits into consideration, respectively. The first neural network is called the Lagrangian network and the second one is called the primal-dual network. In both neural-network-based computation schemes, while the desired accelerations of the end-effector for a specific task are given to the neural networks as their inputs, the signals of the minimum driving joint torques are generated as their outputs to drive the manipulator arm. Both proposed recurrent neural networks are shown to be capable of generating minimum stable driving joint torques. In addition, the driving joint torques computed by the primal-dual network are shown never exceeding the joint torque limits.     

Diagonal recurrent neural networks for dynamic systems control

A new neural paradigm called diagonal recurrent neural network (DRNN) is presented. The architecture of DRNN is a modified model of the fully connected recurrent neural network with one hidden layer, and the hidden layer comprises self-recurrent neurons. Two DRNN's are utilized in a control system, one as an identifier called diagonal recurrent neuroidentifier (DRNI) and the other as a controller called diagonal recurrent neurocontroller (DRNC). A controlled plant is identified by the DRNI, which then provides the sensitivity information of the plant to the DRNC. A generalized dynamic backpropagation algorithm (DBP) is developed and used to train both DRNC and DRNI. Due to the recurrence, the DRNN can capture the dynamic behavior of a system. To guarantee convergence and for faster learning, an approach that uses adaptive learning rates is developed by introducing a Lyapunov function. Convergence theorems for the adaptive backpropagation algorithms are developed for both DRNI and DRNC. The proposed DRNN paradigm is applied to numerical problems and the simulation results are included.     

Novel robust stability criteria of discrete-time stochastic recurrent neural networks with time delay

The problem of robust global exponential stability is investigated for a class of stochastic uncertain discrete-time recurrent neural networks with time delay. In this paper, the midpoint of the time delay's variation interval is introduced, and the variation interval is divided into two subintervals. Then, by constructing a new Lyapunov–Krasovskii functional and checking its variation in the two subintervals, respectively, some novel delay-dependent stability criteria for the addressed neural networks are derived. Numerical examples are provided to show that the achieved conditions are less conservative than some existing ones in the literature.