Seeker optimization algorithm for tuning the structure and parameters of neural networks

Seeker optimization algorithm (SOA) is a novel population-based heuristic stochastic search algorithm, which is based on the concept of simulating the act of human searching. In the SOA, the search direction is determined by seeker’s egotistic behavior, altruistic behavior and pro-activeness behavior, while step length is given by uncertainty reasoning behavior. In this paper, the application of the SOA to tuning the structures and parameters of artificial neural networks (ANNs) is presented as a new evolutionary method of ANN training. Simulation experiments for pattern classification and function approximation are performed. The comparisons of the SOA between BP algorithms and other evolutionary algorithms (EAs) are studied. The simulation results show that the performance of the SOA is better than or, at least, equivalent to that of other EAs (i.e., DE and two variations of PSO) for all the listed problems. Moreover, the ANNs with link switches trained by the SOA can provide better or comparable learning capabilities with much less number of links than ones by BP algorithms (i.e., GDX, RP, OSS and SCG). Hence, SOA can simultaneously tune the structures and the weight values, and, though SOA is more computationally intensive, it is believed that SOA will become a promising candidate for training ANNs.     

Fractional-Neuro-Optimizer: A Neural-Network-Based Optimization Method

Up to now, there have been many attempts in the use of artificial neural networks (ANNs) for solving optimization problems and some types of ANNs, such as Hopfield network and Boltzmann machine, have been applied for combinatorial optimization problems. However, there are some restrictions in the use of ANNs as optimizers. For example: (1) ANNs cannot optimize continuous variable problems; (2) discrete problems should be mapped into the neural networks’ architecture; and (3) most of the existing neural networks are applicable only for a class of smooth optimization problems and global convexity conditions on the objective functions and constraints are required. In this paper, we introduce a new procedure for stochastic optimization by a recurrent ANN. The concept of fractional calculus is adopted to propose a novel weight updating rule. The introduced method is called fractional-neuro-optimizer (FNO). This method starts with an initial solution and adjusts the network’s weights by a new heuristic and unsupervised rule to reach a good solution. The efficiency of FNO is compared to the genetic algorithm and particle swarm optimization techniques. Finally, the proposed FNO is used for determining the parameters of a proportional–integral–derivative controller for an automatic voltage regulator power system and is applied for designing the water distribution networks.     

Artificial neural network training using a new efficient optimization algorithm

Because search space in artificial neural networks (ANNs) is high dimensional and multimodal which is usually polluted by noises and missing data, the process of weight training is a complex continuous optimization problem. This paper deals with the application of a recently invented metaheuristic optimization algorithm, bird mating optimizer (BMO), for training feed-forward ANNs. BMO is a population-based search method which tries to imitate the mating ways of bird species for designing optimum searching techniques. In order to study the usefulness of the proposed algorithm, BMO is applied to weight training of ANNs for solving three real-world classification problems, namely, Iris flower, Wisconsin breast cancer, and Pima Indian diabetes. The performance of BMO is compared with those of the other classifiers. Simulation results indicate the superior capability of BMO to tackle the problem of ANN weight training. BMO is also applied to model fuel cell system which has been addressed as an open and demanding problem in electrical engineering. The promising results verify the potential of BMO algorithm.     

Evolution of neural networks for classification and regression

Although Artificial Neural Networks (ANNs) are importantdata mining techniques, the search for the optimal ANN is a challenging task: the ANN should learn the input–output mapping without overfitting the data and training algorithms may get trapped in local minima. The use of Evolutionary Computation (EC) is a promising alternative for ANN optimization. This work presents two hybrid EC/ANN algorithms: the first evolves neural topologies while the latter performs simultaneous optimization of architectures and weights. Sixteen real-world tasks were used to test these strategies. Competitive results were achieved when compared with a heuristic model selection and other Data Mining algorithms.     

An Improved Particle Swarm Optimization for Evolving Feedforward Artificial Neural Networks

This paper presents a new evolutionary artificial neural network (ANN) algorithm named IPSONet that is based on an improved particle swarm optimization (PSO). The improved PSO employs parameter automation strategy, velocity resetting, and crossover and mutations to significantly improve the performance of the original PSO algorithm in global search and fine-tuning of the solutions. IPSONet uses the improved PSO to address the design problem of feedforward ANN. Unlike most previous studies on only using PSO to evolve weights of ANNs, this study puts its emphasis on using the improved PSO to evolve simultaneously structure and weights of ANNs by a specific individual representation and evolutionary scheme. The performance of IPSONet has been evaluated on several benchmarks. The results demonstrate that IPSONet can produce compact ANNs with good generalization ability.     

A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks

This paper presents a new algorithm, called adaptive merging and growing algorithm (AMGA), in designing artificial neural networks (ANNs). This algorithm merges and adds hidden neurons during the training process of ANNs. The merge operation introduced in AMGA is a kind of a mixed mode operation, which is equivalent to pruning two neurons and adding one neuron. Unlike most previous studies, AMGA puts emphasis on autonomous functioning in the design process of ANNs. This is the main reason why AMGA uses an adaptive not a predefined fixed strategy in designing ANNs. The adaptive strategy merges or adds hidden neurons based on the learning ability of hidden neurons or the training progress of ANNs. In order to reduce the amount of retraining after modifying ANN architectures, AMGA prunes hidden neurons by merging correlated hidden neurons and adds hidden neurons by splitting existing hidden neurons. The proposed AMGA has been tested on a number of benchmark problems in machine learning and ANNs, including breast cancer, Australian credit card assessment, and diabetes, gene, glass, heart, iris, and thyroid problems. The experimental results show that AMGA can design compact ANN architectures with good generalization ability compared to other algorithms.     

Calibrating artificial neural networks by global optimization

Artificial neural networks (ANNs) are used extensively to model unknown or unspecified functional relationships between the input and output of a “black box” system. In order to apply the generic ANN concept to actual system model fitting problems, a key requirement is the training of the chosen (postulated) ANN structure. Such training serves to select the ANN parameters in order to minimize the discrepancy between modeled system output and the training set of observations. We consider the parameterization of ANNs as a potentially multi-modal optimization problem, and then introduce a corresponding global optimization (GO) framework. The practical viability of the GO based ANN training approach is illustrated by finding close numerical approximations of one-dimensional, yet visibly challenging functions. For this purpose, we have implemented a flexible ANN framework and an easily expandable set of test functions in the technical computing system Mathematica. The MathOptimizer Professional global-local optimization software has been used to solve the induced (multi-dimensional) ANN calibration problems.     

Design optimization with back-propagation neural networks

A methodology with back-propagation neural network models is developed to explore the artificial neural nets (ANN) technology in the new application territory of design optimization. This design methodology could go beyond the Hopfield network model, Hopfield and Tank (1985), for combinatorial optimization problems In this approach, pattern classification with back-propagation network, the most demonstrated power of neural networks applications, is utilized to identify the boundaries of the feasible and the infeasible design regions. These boundaries enclose the multi-dimensional space within which designs satisfy all design criteria. A feedforward network is then incorporated to perform function approximation of the design objective function. This approximation is performed by training the feedforward network with objective functions evaluated at selected design sets in the feasible design regions. Additional optimum design sets in the classified feasible regions are calculated and included in the successive training sets to improve the function mapping. Iteration is continued until convergent criteria are satisfied. This paper demonstrates that the artificial neural nets technology provides a global perspective of the entire design space with good and near optimal solutions. ANN can indeed be a potential technology for design optimization.     

Estimation of chemical oxygen demand by ultraviolet spectroscopic profiling and artificial neural networks

A simple method based on the mathematical treatment of spectral absorbance profiles in conjunction with artificial neural networks (ANNs) is demonstrated for rapidly estimating chemical oxygen demand (COD) values of wastewater samples. In order to improve spectroscopic analysis and ANN training time as well as to reduce the storage space of the trained ANN algorithm, it is necessary to decrease the ANN input vector size by extracting unique characteristics from the raw input pattern. Key features from the spectral absorbance pattern were therefore selected to obtain the spectral absorbance profile, reducing the ANN input vector from 160 to 10 selected inputs. The results indicate that the COD values obtained from the selected absorbance profiles agreed well with those obtained from the entire absorbance pattern. The spectral absorbance profile technique was also compared to COD values estimated by a multiple linear regression (MLR) model to validate whether ANNs were better and more robust models for rapid COD analysis. It was found that the ANN model predicted COD values closer to standard COD values than the MLR model.     

A generic fuzzy aggregation operator: rules extraction from and insertion into artificial neural networks

Multilayered feedforward artificial neural networks (ANNs) are black boxes. Several methods have been published to extract a fuzzy system from a network, where the input–output mapping of the fuzzy system is equivalent to the mapping of the ANN. These methods are generalized by means of a new fuzzy aggregation operator. It is defined by using the activation function of a network. This fact lets to choose among several standard aggregation operators. A method to extract fuzzy rules from ANNs is presented by using this new operator. The insertion of fuzzy knowledge with linguistic hedges into an ANN is also defined thanks to this operator.     

Derivative and GA-based methods in metamodeling of back-propagation neural networks for constrained approximate optimization

Artificial neural networks (ANN) have been extensively used as global approximation tools in the context of approximate optimization. ANN traditionally minimizes the absolute difference between target outputs and approximate outputs thereby resulting in approximate optimal solutions being sometimes actually infeasible when it is used as a metamodel for inequality constraint functions. The paper explores the development of the efficient back-propagation neural network (BPN)-based metamodel that ensures the constraint feasibility of approximate optimal solution. The BPN architecture is optimized via two approaches of both derivative-based method and genetic algorithm (GA) to determine interconnection weights between layers in the network. The verification of the proposed approach is examined by adopting a standard ten-bar truss problem. Finally, a GA-based approximate optimization of suspension with an optical flying head is conducted to enhance the shock resistance capability in addition to dynamic characteristics.     

Distributed memory systems for simulating artificial neural networks

In executing tasks involving intelligent information processing, the human brain performs better than the digital computer. The human brain derives its power from a large number [O(10¹¹)] of neurons which are interconnected by a dense interconnection network [O(10⁵) connections per neuron]. Artificial neural network (ANN) paradigms adopt the structure of the brain to try to emulate the intelligent information processing methods of the brain. ANN techniques are being employed to solve problems in areas such as pattern recognition, and robotic processing. Simulation of ANNs involves implementation of large number of neurons and a massive interconnection network. In this paper, we discuss various simulation models of ANNs and their implementation on distributed memory systems. Our investigations reveal that communication-efficient networks of distributed memory systems perform better than other topologies in implementing ANNs.     

Global optimization of a neural network-hidden Markov model hybrid

The integration of multilayered and recurrent artificial neural networks (ANNs) with hidden Markov models (HMMs) is addressed. ANNs are suitable for approximating functions that compute new acoustic parameters, whereas HMMs have been proven successful at modeling the temporal structure of the speech signal. In the approach described, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters. Results on speaker-independent recognition experiments using this integrated ANN-HMM system on the TIMIT continuous speech database are reported.     

A hybrid surrogate and pattern search optimization method and application to microelectronics

A hybrid method for robust and efficient optimization process is developed by integrating a new response surface method and pattern search algorithm. The method is based on: (1) multipoint approximations of the objective and constraint functions, (2) a multiquadric radial basis function (RBF) for the zeroth-order function approximation and a new RBF plus polynomial-based moving least-squares approximation for the first-order enhanced function approximation, and (3) a pattern search algorithm to impose a descent condition and applied adaptive subregion management strategy. Several numerical examples are presented to illustrate accuracy and computational efficiency of the proposed method for both function approximation and design optimization. To demonstrate the effectiveness of the proposed hybrid method, it is applied to obtain optimum designs of a microelectronic packaging system. A two-stage optimization approach is proposed for the design optimization. The material properties of microelectronic packaging system and the shape parameters of solder ball are selected as design variables. Through design optimization, significant improvements of durability performances are obtained using the proposed hybrid optimization method.     

Estimation of heat transfer in oscillating annular flow using artifical neural networks

In this study, the prediction of heat transfer from a surface having constant heat flux subjected to oscillating annular flow is investigated using artificial neural networks (ANNs). An experimental study is carried out to estimate the heat transfer characteristics as a function of some input parameters, namely frequency, amplitude, heat flux and filling heights. In the experiments, a piston cylinder mechanism is used to generate an oscillating flow in a liquid column at certain frequency and amplitude. The cycle-averaged values are considered in the calculation of heat transfer using the control volume approach. An experimentally evaluated data set is prepared to be processed with the use of neural networks. Back propagation algorithm, the most common learning method for ANNs, is used for training and testing the network. Results of the experiments and the ANN are in close agreements with errors less than 5%. The study showed that the ANNs could be used effectively for modeling oscillating flow heat transfer in a vertical annular duct.     

Neural identification of dynamic systems on FPGA with improved PSO learning

This work introduces hardware implementation of artificial neural networks (ANNs) with learning ability on field programmable gate array (FPGA) for dynamic system identification. The learning phase is accomplished by using the improved particle swarm optimization (PSO). The improved PSO is obtained by modifying the velocity update function. Adding an extra term to the velocity update function reduced the possibility of stucking in a local minimum. The results indicates that ANN, trained using improved PSO algorithm, converges faster and produces more accurate results with a little extra hardware utilization cost.     

Artificial neural network dexterous robotics hand optimal control methodology: grasping and manipulation forces optimization

Optimal fingertip forces can always be computed through the well-known optimization algorithms. However, computation time has always remained a real-time constraint. This article presents an efficient scheme to compute optimal grasping and manipulation forces for dexterous robotics hands. This is expressed as a quadratic optimization problem, and an artificial neural network (ANN) is used to learn such quadratic optimization formulations. Computation has been based on a nonlinear model of fingertip contacts and slips. In achieving object grasping while in motion, the hand Jacobian is considered an important matrix to be computed, but it is also highly intensive for real-time computed applications. Consequently, we investigated an efficient approach using artificial neural networks to learn optimal grasping forces. An ANN is used here to learn the optimal contact forces relating hand joint-space torques to the resulting object force. The results have indicated that the ANN has reduced computation times to reasonable values owing to its ability to map nonlinear force relations. Furthermore, the results have revealed that ANNs are capable of learning highly nonlinear relations relating to distributed fingertip forces and joint torques. The technique developed has also proved to be suitable for off-line learning of computed fingertip forces, even with large training samples.     

AN ANALYSIS OF NEURAL NETWORKS AS SIMULATORS AND EMULATORS

Recent advances in artificial neural networks (ANNs) have led to the design and construction of neuroarchitectures as simulator and emulators of a variety of problems in science and engineering. Such problems include pattern recognition, prediction, optimization, associative memory, and control of dynamic systems. This paper offers an analytical overview of the most successful design, implementation, and application of neuroarchitectures as neurosimulators and neuroemulators. It also outlines historical notes on the formulation of basic biological neuron, artificial computational models, network architectures, and learning processes of the most common ANN; describes and analyzes neurosimulation on parallel architecture both in software and hardware (neurohardware); presents the simulation of ANNs on parallel architectures; gives a brief introduction of ANNs in vector microprocessor systems; and presents ANNs in terms of the "new technologies". Specifically, it discusses cellular computing, cellular neural networks (CNNs), a new proposition for unsupervised neural networks (UNNs), and pulse coupled neural networks (PCNNs).     

The ANNS approach to DEM reconstruction

This research has 6 fundamental aims: (i) to present a modified version of Taylor's interpolation, one that is more effective and faster than the original; (ii) outline the capability of artificial neural networks (ANNs) to perform an optimal functional approximation of the digital elevation model reconstruction from a satellite map, using a small and independent sample of Global Positioning System observations; (iii) demonstrate experimentally how ANNs outperform the traditional and most used algorithm for the height interpolation (Taylor's interpolation); (iv) introduce a new ANN, the Conic Net, able to outperform the results of the classic and more known multilayer perceptron; (v) determine that Conic Nets, even when using Taylor's modified interpolation as input features, are able to optimally approximate the heights with one order of magnitude more than the original satellite map; and (vi) make evident the possibility to interpolate the DEM heights through an ANN, which learns a data set of known points.     

A novel neural network for the synthesis of antennas and microwave devices

A novel artificial neural network (SYNTHESIS-ANN) is presented, which has been designed for computationally intensive problems and applied to the optimization of antennas and microwave devices. The antenna example presented is optimized with respect to voltage standing-wave ratio, bandwidth, and frequency of operation. A simple microstrip transmission line problem is used to further describe the ANN effectiveness, in which microstrip line width is optimized with respect to line impedance. The ANNs exploit a unique number representation of input and output data in conjunction with a more standard neural network architecture. An ANN consisting of a heteroassociative memory provided a very efficient method of computing necessary geometrical values for the antenna when used in conjunction with a new randomization process. The number representation used provides significant insight into this new method of fault-tolerant computing. Further work is needed to evaluate the potential of this new paradigm.