AN ANALYSIS OF NEURAL NETWORKS AS SIMULATORS AND EMULATORS

Recent advances in artificial neural networks (ANNs) have led to the design and construction of neuroarchitectures as simulator and emulators of a variety of problems in science and engineering. Such problems include pattern recognition, prediction, optimization, associative memory, and control of dynamic systems. This paper offers an analytical overview of the most successful design, implementation, and application of neuroarchitectures as neurosimulators and neuroemulators. It also outlines historical notes on the formulation of basic biological neuron, artificial computational models, network architectures, and learning processes of the most common ANN; describes and analyzes neurosimulation on parallel architecture both in software and hardware (neurohardware); presents the simulation of ANNs on parallel architectures; gives a brief introduction of ANNs in vector microprocessor systems; and presents ANNs in terms of the "new technologies". Specifically, it discusses cellular computing, cellular neural networks (CNNs), a new proposition for unsupervised neural networks (UNNs), and pulse coupled neural networks (PCNNs).     

Auditory learning: a developmental method

Motivated by the human autonomous development process from infancy to adulthood, we have built a robot that develops its cognitive and behavioral skills through real-time interactions with the environment. We call such a robot a developmental robot. In this paper, we present the theory and the architecture to implement a developmental robot and discuss the related techniques that address an array of challenging technical issues. As an application, experimental results on a real robot, self-organizing, autonomous, incremental learner (SAIL), are presented with emphasis on its audition perception and audition-related action generation. In particular, the SAIL robot conducts the auditory learning from unsegmented and unlabeled speech streams without any prior knowledge about the auditory signals, such as the designated language or the phoneme models. Neither available before learning starts are the actions that the robot is expected to perform. SAIL learns the auditory commands and the desired actions from physical contacts with the environment including the trainers.     

A non-symbolic implementation of abdominal pain estimation in childhood

The abdominal pain is a very common disease in childhood, which lurks complications. Pediatric surgeons have to estimate at least 15 clinical and laboratory factors in order to make a diagnosis and decide about performing a surgical operation of the abdomen. Artificial Neural Networks (ANNs) are particular implementations of Artificial Intelligence (AI) systems and they are used in a wide area of application fields. This study examines the implementation of ANN architectures, using Multi-Layer Perceptron (MLP) neural networks and Probabilistic Neural Networks (PNN) architectures, in order to specify the appropriate ANN structure for abdominal pain estimation in childhood. The architecture with the best performance is a fully interconnected MLP neural network with an input layer of 15 nodes, one hidden layer of 5 neurons and an output layer, with error back-propagation algorithm being used as the learning scheme. In the output layer, the estimation of appendicitis’ stage is reached automatically. The proposed ANN achieved a percentage of 88.5% of correct classification on testing set cases. Further analysis of obtained results, exhibited the ability of ANN for distinguishing the necessity of a case for operative treatment of abdominal pain based on diagnostic features, attaining a percentage of 100% of successful prognosis over the cases of testing set. The aim of proposed MLP neural network is to assist surgeons in appendicitis prediction, avoiding an unnecessary operative treatment.     

An offset error compensation method for improving ANN accuracy when used for position control of precision machinery

Artificial Neural Networks (ANNs) have recently become the focus of considerable attention in many disciplines, including robot control, where they can be used as a general class of nonlinear models to solve highly nonlinear control problems. Feedforward neural networks have been widely applied for modelling and control purposes. One of the ANN applications in robot control is for the solution of the inverse kinematic problem, which is important in path planning of robot manipulators. This paper proposes an iterative approach and an offset error compensation method to improve the accuracy of the inverse kinematic solutions by using an ANN and a forward kinematic model of a robot. The offset error compensation method offers potential to generate accurately the inverse solution for a class of problems which have an easily obtained forward model and a complicated solution.Now Lecturing in Taiwan.     

A time-multiplexed reconfigurable neuroprocessor

Built around biological-like information-processing structures, artificial neural networks (ANNs) have demonstrated their power and usefulness in areas where identification and adaptability are most crucial. After sufficient training with a given set of problem data (using an arbitrary learning rule), ANNs can independently form internal representations (models) of the data's underlying phenomenon. ANNs typically have a large number of highly interconnected processing elements, which constrains their hardware implementation and network architecture. To obtain high density and processing-element connectivity, most ANN architectures employ some kind of resource sharing. A multilayer structure allows processing elements to share communication lines and control circuitry, so we chose to develop a multilayered neuroprocessor based on pipelined time-step interneural communication. Other pulse stream architectures use the rate of impulses to represent the neural states. Our architecture, however, represents the neural states through pulse amplitude modulation. Also, in our design, the analog computation consists of integrating the bipolar input pulses corresponding to the excitatory and inhibitory activations. The processing-element analog path consists. of CMOS transmission gates controlled by buffered signals originating from the neuroprocessor control unit. The processing elements broadcast their output states, held on a local capacitor, during their assigned time slots. These features are desirable to meet the design goals of versatility, high density, high connectivity, and scalability     

A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks

This paper presents a new algorithm, called adaptive merging and growing algorithm (AMGA), in designing artificial neural networks (ANNs). This algorithm merges and adds hidden neurons during the training process of ANNs. The merge operation introduced in AMGA is a kind of a mixed mode operation, which is equivalent to pruning two neurons and adding one neuron. Unlike most previous studies, AMGA puts emphasis on autonomous functioning in the design process of ANNs. This is the main reason why AMGA uses an adaptive not a predefined fixed strategy in designing ANNs. The adaptive strategy merges or adds hidden neurons based on the learning ability of hidden neurons or the training progress of ANNs. In order to reduce the amount of retraining after modifying ANN architectures, AMGA prunes hidden neurons by merging correlated hidden neurons and adds hidden neurons by splitting existing hidden neurons. The proposed AMGA has been tested on a number of benchmark problems in machine learning and ANNs, including breast cancer, Australian credit card assessment, and diabetes, gene, glass, heart, iris, and thyroid problems. The experimental results show that AMGA can design compact ANN architectures with good generalization ability compared to other algorithms.     

A neuro-emulator with embedded capabilities for generalized learning

Artificial neural networks (ANN) are being used as one of the prime computing tool for an increasing number of applications. This is due in part to the ANN's ability to adapt to changes via learning. The dynamic nature of many applications as well as the computational and storage requirements of current learning algorithms creates a need for high performance neuro-architectures with learning capabilities. In this paper we identify a set of computational, communication and storage requirements for learning in ANNs. These requirements are representative of a wide variety of algorithms for different learning approaches. We propose a novel neuro-emulator that provides the computational ability for the stated requirements. While meeting all the identified requirements the new architecture maintains a high performance during learning. To show the capabilities of the proposed machine we present four diverse learning algorithms and step through the execution of each using the proposed architecture. We include an evaluation of the machine performance as well as a comparison with other architectures. It is shown that with a modest amount of hardware the proposed architecture yields an extremely high number of connections per second.     

Supervised fuzzy reinforcement learning for robot navigation

This paper addresses a new method for combination of supervised learning and reinforcement learning (RL). Applying supervised learning in robot navigation encounters serious challenges such as inconsistent and noisy data, difficulty for gathering training data, and high error in training data. RL capabilities such as training only by one evaluation scalar signal, and high degree of exploration have encouraged researchers to use RL in robot navigation problem. However, RL algorithms are time consuming as well as suffer from high failure rate in the training phase. Here, we propose Supervised Fuzzy Sarsa Learning (SFSL) as a novel idea for utilizing advantages of both supervised and reinforcement learning algorithms. A zero order Takagi–Sugeno fuzzy controller with some candidate actions for each rule is considered as the main module of robot's controller. The aim of training is to find the best action for each fuzzy rule. In the first step, a human supervisor drives an E-puck robot within the environment and the training data are gathered. In the second step as a hard tuning, the training data are used for initializing the value (worth) of each candidate action in the fuzzy rules. Afterwards, the fuzzy Sarsa learning module, as a critic-only based fuzzy reinforcement learner, fine tunes the parameters of conclusion parts of the fuzzy controller online. The proposed algorithm is used for driving E-puck robot in the environment with obstacles. The experiment results show that the proposed approach decreases the learning time and the number of failures; also it improves the quality of the robot's motion in the testing environments.     

Learning long-term dependencies in NARX recurrent neural networks

It has previously been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show that the long-term dependencies problem is lessened for a class of architectures called nonlinear autoregressive models with exogenous (NARX) recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks. We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on long-term dependency problems. We also describe in detail some of the assumptions regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions.     

Speeding up backpropagation using multiobjective evolutionary algorithms

The use of backpropagation for training artificial neural networks (ANNs) is usually associated with a long training process. The user needs to experiment with a number of network architectures; with larger networks, more computational cost in terms of training time is required. The objective of this letter is to present an optimization algorithm, comprising a multiobjective evolutionary algorithm and a gradient-based local search. In the rest of the letter, this is referred to as the memetic Pareto artificial neural network algorithm for training ANNs. The evolutionary approach is used to train the network and simultaneously optimize its architecture. The result is a set of networks, with each network in the set attempting to optimize both the training error and the architecture. We also present a self-adaptive version with lower computational cost. We show empirically that the proposed method is capable of reducing the training time compared to gradient-based techniques.     

An alternative approach for modelling and simulation of traffic data: artificial neural networks

It is assumed that there is a complicated relationship between the driver characteristics and involvement in traffic accidents. It is quite difficult to simulate the effects of these driver characteristics into the traffic accidents. The artificial neural networks (ANN) approach is proposed for training-predicting the database in this paper since it is a more flexible and assumption-free methodology. The networks are organised in different architectures and the results have been compared in order to determine the best fitting one. Finally, the best possible architecture is selected for a better representation of the survey data and the prediction of accident percentage. The predictions about the outputs for the inputs which are not used in the training of the ANN provide information about the drivers which cannot be reached in the database. The predictions are highly satisfactory and the ANNs have been found to be reliable processing systems for modelling and simulation in the traffic data assessments.     

Performance of artificial neural networks in nearshore wave power prediction

In this paper the assessment of the wave energy potential in nearshore coastal areas is investigated by means of artificial neural networks (ANNs). The performance of the ANNs is compared with in situ measurements and spectral numerical modelling (the conventional tool for wave energy assessment). For this purpose, 13 years of records of two buoys, one offshore and one inshore, with an hourly frequency are used to develop an ANN model for predicting the nearshore wave power. The best suited architecture was selected after assessing the performance of 480 ANN models involving twelve different architectures. The results predicted by the ANN model were compared with the measured data and those obtained by means of the SWAN (Simulating Waves Nearshore) spectral model. The quality in the predictions of the ANN model shows that this type of artificial intelligence models constitutes a powerful tool to forecast the wave energy potential at particular coastal site with great accuracy, and one that overcomes some of the disadvantages of the conventional tools for nearshore wave power prediction.     

Methods used for the development of neural networks for the prediction of water resource variables in river systems: Current status and future directions

Over the past 15 years, artificial neural networks (ANNs) have been used increasingly for prediction and forecasting in water resources and environmental engineering. However, despite this high level of research activity, methods for developing ANN models are not yet well established. In this paper, the steps in the development of ANN models are outlined and taxonomies of approaches are introduced for each of these steps. In order to obtain a snapshot of current practice, ANN development methods are assessed based on these taxonomies for 210 journal papers that were published from 1999 to 2007 and focus on the prediction of water resource variables in river systems. The results obtained indicate that the vast majority of studies focus on flow prediction, with very few applications to water quality. Methods used for determining model inputs, appropriate data subsets and the best model structure are generally obtained in an ad-hoc fashion and require further attention. Although multilayer perceptrons are still the most popular model architecture, other model architectures are also used extensively. In relation to model calibration, gradient based methods are used almost exclusively. In conclusion, despite a significant amount of research activity on the use of ANNs for prediction and forecasting of water resources variables in river systems, little of this is focused on methodological issues. Consequently, there is still a need for the development of robust ANN model development approaches.     

Pruned Feed-forward Networks for Efficient Implementation of Multiple FIR Filters with Arbitrary Frequency Responses

A new algorithm is presented for efficient implementation of multiple FIR filters in real-time applications. We introduce an analogy between the multiple FIR filters and linear feed-forward networks, and show how the FIR filters with any frequency characteristics may be designed by a learning algorithm of the network with proper choice of training patterns. Starting from a fully-connected feed-forward architecture, more efficient network architectures may be obtainable by pruning connection weights with minor contributions. For demonstration we design feed-forward networks for 16 bandpass cochlear filters with much less connection weights and moderate performance degradation.     

Neural network method: delay and system of delay differential equations

In the present article, delay and system of delay differential equations are treated using feed-forward artificial neural networks. We have solved multiple problems using neural network architectures with different depths. The neural networks are trained using the extreme learning machine algorithm for the satisfaction of delay differential equations and associated initial/boundary conditions. Further, numerical rates of convergence of the proposed algorithm are reported based on variation of error in the obtained solution for different number of training points. Emphasis is on analysing whether deeper network architectures trained with extreme learning machine algorithm can perform better than shallow network architectures for approximating the solutions of delay differential equations.

     

Computational capabilities of recurrent NARX neural networks

Recently, fully connected recurrent neural networks have been proven to be computationally rich-at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. These networks are based upon Nonlinear AutoRegressive models with eXogenous Inputs (NARX models), and are therefore called NARX networks. As opposed to other recurrent networks, NARX networks have a limited feedback which comes only from the output neuron rather than from hidden states. They are formalized by y(t)=Psi(u(t-n(u)), ..., u(t-1), u(t), y(t-n(y)), ..., y(t-1)) where u(t) and y(t) represent input and output of the network at time t, n(u) and n(y) are the input and output order, and the function Psi is the mapping performed by a Multilayer Perceptron. We constructively prove that the NARX networks with a finite number of parameters are computationally as strong as fully connected recurrent networks and thus Turing machines. We conclude that in theory one can use the NARX models, rather than conventional recurrent networks without any computational loss even though their feedback is limited. Furthermore, these results raise the issue of what amount of feedback or recurrence is necessary for any network to be Turing equivalent and what restrictions on feedback limit computational power.     

Artificial neural networks: a tutorial

Artificial neural nets (ANNs) are massively parallel systems with large numbers of interconnected simple processors. The article discusses the motivations behind the development of ANNs and describes the basic biological neuron and the artificial computational model. It outlines network architectures and learning processes, and presents some of the most commonly used ANN models. It concludes with character recognition, a successful ANN application     

Fractional snow cover mapping through artificial neural network analysis of MODIS surface reflectance

Accurate areal measurements of snow cover extent are important for hydrological and climate modeling. The traditional method of mapping snow cover is binary where a pixel is considered either snow-covered or snow-free. Fractional snow cover (FSC) mapping can achieve a more precise estimate of areal snow cover extent by estimating the fraction of a pixel that is snow-covered. The most common snow fraction methods applied to Moderate Resolution Imaging Spectroradiometer (MODIS) images have been spectral unmixing and an empirical Normalized Difference Snow Index (NDSI). Machine learning is an alternative for estimating FSC as artificial neural networks (ANNs) have been successfully used for estimating the subpixel abundances of other surfaces. The advantages of ANNs are that they can easily incorporate auxiliary information such as land cover type and are capable of learning nonlinear relationships between surface reflectance and snow fraction. ANNs are especially applicable to mapping snow cover extent in forested areas where spatial mixing of surface components is nonlinear. This study developed a multilayer feed-forward ANN trained through backpropagation to estimate FSC using MODIS surface reflectance, NDSI, Normalized Difference Vegetation Index (NDVI) and land cover as inputs. The ANN was trained and validated with higher spatial-resolution FSC maps derived from Landsat Enhanced Thematic Mapper Plus (ETM+) binary snow cover maps. Testing of the network was accomplished over training and independent test areas. The developed network performed adequately with RMSE of 12% over training areas and slightly less accurately over the independent test scenes with RMSE of 14%. The developed ANN also compared favorably to the standard MODIS FSC product. The study also presents a comprehensive validation of the standard MODIS snow fraction product whose performance was found to be similar to that of the ANN.     

Neural network implementation using bit streams

A new method for the parallel hardware implementation of artificial neural networks (ANNs) using digital techniques is presented. Signals are represented using uniformly weighted single-bit streams. Techniques for generating bit streams from analog or multibit inputs are also presented. This single-bit representation offers significant advantages over multibit representations since they mitigate the fan-in and fan-out issues which are typical to distributed systems. To process these bit streams using ANNs concepts, functional elements which perform summing, scaling, and squashing have been implemented. These elements are modular and have been designed such that they can be easily interconnected. Two new architectures which act as monotonically increasing differentiable nonlinear squashing functions have also been presented. Using these functional elements, a multilayer perceptron (MLP) can be easily constructed. Two examples successfully demonstrate the use of bit streams in the implementation of ANNs. Since every functional element is individually instantiated, the implementation is genuinely parallel. The results clearly show that this bit-stream technique is viable for the hardware implementation of a variety of distributed systems and for ANNs in particular.