Effect of complexity on learning ability of recurrent neural networks

This paper demonstrates that recurrent neural networks can be used effectively to estimate unknown, complicated nonlinear dynamics. The emphasis of this paper is on the distinguishable properties of dynamics at the edge of chaos, i.e., between ordered behavior and chaotic behavior. We introduce new stochastic parameters, defined as combinations of standard parameters, and reveal relations between these parameters and the complexity of the network dynamics by simulation experiments. We then propose a novel learning method whose core is to keep the complexity of the network dynamics to the dynamics phase which has been distinguished using formulations of the experimental relations. In this method, the standard parameters of neurons are changed by the core part and also according to the global error measure calculated by the well-known simple back-propagation algorithm. Some simulation studies show that the core part is effective for recurrent neural network learning, and suggest the existence of excellent learning ability at the edge of chaos. This work was presented, in part, at the Second International Symposium on Artificial Life and Robotics, Oita, Japan, February 18–20, 1997     

Aspects of neural modeling of multidimensional chaotic attractors

This article discusses various aspects of neural modeling of multidimensional chaotic attractors. The Lorenz and Rosler attractors are considered as representative cases and are thoroughly examined. These two dynamical systems are expressed, within acceptable accuracy limits, by the corresponding systems of difference equations. Initially, the complete neural models of the attractors are examined. In this case, the neural networks are supplied with the values X_n , Y_n , Z_n of the systems to predict all the next components (X_{n +1}, Y_{n +1}, and Z_{n +1}) of the attractors. In the second case, named ‘component simulation’, the neural models are trained to predict only one of the values X_n , Y_n , Z_n , when they are fed with the complete input vector as in the first case. In the third case, the proposed neural networks are trained to predict only one component (X_{n +1}, Y_{n +1}, or Z_{n +1}) of the attractors, given a number of past values of the same component. Finally, the ability of the networks to predict the Y and Z components of an X time series of the dynamical systems is examined. Since the response of some networks is not satisfactory, the distribution of absolute error is considered in order to form a realistic picture of the networks’ performance.     

Modeling spiking neural networks

A notation for the functional specification of a wide range of neural networks consisting of temporal or non-temporal neurons, is proposed. The notation is primarily a mathematical framework, but it can also be illustrated graphically and can be extended into a language in order to be automated. Its basic building blocks are processing entities, finer grained than neurons, connected by instant links, and as such they form sets of interacting entities resulting in bigger and more sophisticated structures. The hierarchical nature of the notation supports both top-down and bottom-up specification approaches. The use of the notation is evaluated by a detailed example of an integrated tangible agent consisting of sensors, a computational part, and actuators. A process from specification to both software and hardware implementation is proposed.     

Estimation of the dimension of chaotic dynamical systems using neural networks and robust location estimate

In this paper, we propose a new algorithm for the estimation of the dimension of chaotic dynamical systems using neural networks and robust location estimate.The basic idea is that a member of a time series can be optimally expressed as a deterministic function of the d past series values, where d is the dimension of the system. Moreover the neural networks’ learning ability is improved rapidly when the appropriate amount of information is provided to a neural structure which is as complex as needed.To estimate the dimension of a dynamical system, neural networks are trained to learn the component of the attractor expressed by a reconstructed vector in a suitable phase space whose embedding dimension m, has been estimated using the method of mutual information.     

Cellular neural networks to explore complexity

 In this paper the fundamentals of Cellular Neural Networks (CNNs) are introduced. Subsequently it is shown that, due to their locally distributed way of exchanging signals, such structures can be used as powerful devices to simulate and to reproduce, in an analog fashion and low cost, complex behaviors, i.e. dynamics commonly encountered in living systems, such as autonomous wave formation and propagation as well as morphogenetical pattern development. In fact it is proven that both of these behaviours can be simulated with CNNs with the same cell structure, and the thoroughly different dynamics can arise only suitably modulating the CNN cell parameters. Therefore a unifying approach to pattern formation and active wave propagation phenomena is presented. The derivation of the complex phenomena is analytically addressed and several simulation results are also reported. Received: 3 March 1997 / Accepted: 8 April 1997     

Modeling workpiece vibrations with neural networks

The entire workpiece on a lathe vibrates when it is excited at a single point. Frequency and time-domain/time-series techniques can estimate the force-displacement relationships between excitation and the individual points on the workpiece. In this paper, the use of single neural network is proposed to represent the force-displacement relationship between the applied excitation force and the vibration of the whole workpiece. The accuracy of the proposed approach is evaluated on the experimental data. Also, another neural network is used to store the frequency response characteristics of the workpiece.     

Beyond particle systems: Operator networks

Particle systems are used for simulating non-linear dynamics of complex systems. They are computationally attractive, because the models are simple difference equations. The difference equations, however, constitute a closed system lacking scalability and intentionality; it is hard to “reverse engineer” the equations, to understand the relations of the variables and coefficients to the dynamics displayed by the simulation. Consequently, much of the modeling work goes into finding workarounds. In this paper, we study a potential solution. As the main contribution, we formalize particle system computations as mathematical operator networks, to gain intentionality and modularity. Operators also support the inclusion of processes outside the mathematical domain of difference equations. We illustrate the use of operator networks by simulating the construction and dynamics of an hourglass.     

Traceable content protection based on chaos and neural networks

In this paper, a media content encryption/decryption algorithm is designed based on a chaos system and neural networks, which generates random sequences with chaos, and encrypts or decrypts media contents with neural networks in a parallel way. In this scheme, different decryption keys can be used to recover the media content into different copies. That is, the decryption operation gets the content containing certain random sequence that can be used as the identification. With respect to this property, the scheme is used for secure content distribution. Taking the audio content for example, it is encrypted by a key at the sender side and decrypted by different keys at the receiver side. The differences between decryption keys lead to different decrypted audio copies. If one customer distributes his copy to other unauthorized customers, the chaotic sequence contained in the copy can tell the illegal customer. The performances, including security, imperceptibility and robustness, are analyzed, and some experimental results are given to show the scheme's practicability.     

Design of artificial neural networks for tool wear monitoring

An on-line scheme for tool wear monitoring using artificial neural networks (ANNs) has been proposed. Cutting velocity, feed, cutting force and machining time are given as inputs to the ANN, and the flank wear is estimated using the ANN. Different ANN structures are designed and investigated to estimate the tool wear accurately. An existing analytical model is used to obtain the data for various cutting conditions in order to eliminate the huge cost and time associated with generation of training and evaluation data. Motivated by the fact that the tool wear at a given instance of time depends on the tool wear value at a previous instance of time, memory is included in the ANN. ANNs without memory, with one-phase memory, and with two-phase memory are investigated in this study. The effect of various training parameters, such as learning coefficient, momentum, temperature, and number of hidden neurons, on these architectures is studied. The findings and experience obtained should facilitate the design and implementation of reliable and economical real-time systems for tool wear monitoring and identification in intelligent manufacturing.     

Chaotic neural networks and chaotic annealing

A simple model of single neuron with chaotic dynamics is proposed. Neural networks coupled by such neurons have the property of temporal retrieval of stored patterns in a chaotic way. The network is also studied from the viewpoint of optimization. A chaotic annealing technique is developed to search for the global minima of the energy with transient chaos.     

A practical overview of neural networks

This paper overviews the myths and misconceptions that have surrounded neural networks in recent years. Focusing on backpropagation and the Hopfield network, we discuss the problems that have plagued practical application of these techniques, and review some of the recent progress made. Both real and perceived inadequacies of backpropagation are discussed, as well as the need for an understanding of statistics and of the problem domain in order to apply and assess the neural network properly. We consider alternatives or variants to backpropagation, which overcome some of its real limitations. The Hopfield network's poor performance on the traveling salesman problem in combinatorial optimization has colored its reception by engineers; we describe both new research in this area and promising results in other practical optimization applications. Overall, it is hoped, this paper will aid in a more balanced understanding of neural networks. They seem worthy of consideration in many applications, but they do not deserve the status of a panacea – nor are they as fraught with problems as would now seem to be implied.     

Exponential stability of hybrid stochastic neural networks with mixed time delays and nonlinearity

This paper is concerned with the problem of robust exponential stability for a class of hybrid stochastic neural networks with mixed time-delays and Markovian jumping parameters. In this paper, free-weighting matrices are employed to express the relationship between the terms in the Leibniz–Newton formula. Based on the relationship, a linear matrix inequality (LMI) approach is developed to establish the desired sufficient conditions for the mixed time-delays neural networks with Markovian jumping parameters. Finally, two simulation examples are provided to demonstrate the effectiveness of the results developed.     

Decentralized control of a class of large-scale nonlinear systems using neural networks

This paper designs a decentralized neural network (NN) controller for a class of nonlinear large-scale systems, in which strong interconnections are involved. NNs are used to handle unknown functions. The proposed scheme is proved guaranteeing the boundedness of the closed-loop subsystems using only local feedback signals.     

Modeling and prediction with a class of time delay dynamic neural networks

In this paper, we propose a time delay dynamic neural network (TDDNN) to track and predict a chaotic time series systems. The application of artificial neural networks to dynamical systems has been constrained by the non-dynamical nature of popular network architectures. Many of the drawbacks caused by the algebraic structures can be overcome with TDDNNs. TDDNNs have time delay elements in their states. This approach provides the natural properties of physical systems. The minimization of a quadratic performance index is considered for trajectory tracking applications. Gradient computations are presented based on adjoint sensitivity analysis. The computational complexity is significantly less than direct method, but it requires a backward integration capability. We used Levenberg–Marquardt parameter updating method.     

Application of neural networks to temperature control in thermal power plants

In a thermal power plant with once-through boilers, it is important to control the temperature at the middle point where water becomes steam. However, there are many problems in the design of such a control system, due to a long system response delay, dead-zone and saturation of the actuator mechanisms, uncertainties in the system model and/or parameters, and process noise. To overcome these problems, an adaptive controller has been designed using neural networks, and tested extensively via simulations.

One of the key problems in designing such a controller is to develop an efficient training algorithm. Neural networks are usually trained using the output errors of the network, instead of using the output errors of the controlled plant. However, when a neural network is used to control a plant directly, the output errors of the network are unknown, since the desired control actions are unknown. This paper proposes a simple training algorithm for a class of nonlinear systems, which enables the neural network to be trained with the output errors of the controlled plant. The only a priori knowledge of the controlled plant is the direction of its output response. Due to its simple structure and algorithm, and good performance, the proposed controller has high potential for handling difficult problems in process-control systems.     

Applications of neural networks for coordinate transformations in robotics

The use of artificial neural networks is investigated for application to trajectory control problems in robotics. The relative merits of position versus velocity control is considered and a control scheme is proposed in which neural networks are used as static maps (trained off-line) to compute the inverse of the manipulator Jacobian matrix. A proof of the stability of this approach is offered, assuming bounded errors in the static map. A representative two-link robot is investigated using an artificial neural network which has been trained to compute the components of the inverse of the Jacobian matrix. The controller is implemented in the laboratory and its performance compared to a similar controller with the analytical inverse Jacobian matrix.     

Optimising memory usage in n-tuple neural networks

The use of n-tuple or weightless neural networks as pattern recognition devices is well known (Aleksander and Stonham, 1979). They have some significant advantages over the more common and biologically plausible networks, such as multi-layer perceptrons; for example, n-tuple networks have been used for a variety of tasks, the most popular being real-time pattern recognition, and they can be implemented easily in hardware as they use standard random access memories.

In operation, a series of images of an object are shown to the network, each being processed suitably and effectively stored in a memory called a discriminator. Then, when another image is shown to the system, it is processed in a similar manner and the system reports whether it recognises the image; is the image sufficiently similar to one already taught?

If the system is to be able to recognise and discriminate between m-objects, then it must contain m-discriminators. This can require a great deal of memory.

This paper describes various ways in which memory requirements can be reduced, including a novel method for multiple discriminator n-tuple networks used for pattern recognition. By using this method, the memory normally required to handle m-objects can be used to recognise and discriminate between 2^m — 2 objects.     

The impact of neural networks in finance

The financial industry is becoming more and more dependent on advanced computer technologies in order to maintain competitiveness in a global economy. Neural networks represent an exciting technology with a wide scope for potential applications, ranging from routine credit assessment operations to driving of large scale portfolio management strategies. Some of these applications have already resulted in dramatic increases in productivity. This paper brings together, from diverse sources, a collection of current research issues on neural networks in the financial domain. It examines a range of neural network systems related to financial applications from different levels of maturity to fielded products. It discusses the success rate of the neural network systems, and their performance in resolving particular financial problems.     

Parallel processing, neural networks and genetic algorithms

In an earlier paper[1] some recent developments in computational technology to structural engineering were described. The developments included: parallel and distributed computing; neural networks; and genetic algorithms. In this paper, the authors concentrate on parallel implementations of neural networks and genetic algorithms. In the final section of the paper the authors show how a parallel finite element analysis may be undertaken in an efficient manner by preprocessing of the finite element model using a genetic algorithm utilizing a neural network predictor. This preprocessing is the partitioning of the finite element mesh into sub-domains to ensure load balancing and minimum interprocessor communication during the parallel finite element analysis on a MIMD distributed memory computer. © 1998 Published by Elsevier Science Limited. All rights reserved.