Evolutionary artificial neural networks: a review

This paper reviews the use of evolutionary algorithms (EAs) to optimize artificial neural networks (ANNs). First, we briefly introduce the basic principles of artificial neural networks and evolutionary algorithms and, by analyzing the advantages and disadvantages of EAs and ANNs, explain the advantages of using EAs to optimize ANNs. We then provide a brief survey on the basic theories and algorithms for optimizing the weights, optimizing the network architecture and optimizing the learning rules, and discuss recent research from these three aspects. Finally, we speculate on new trends in the development of this area.     

A review of genetic algorithms applied to training radial basis function networks

The problems associated with training feedforward artificial neural networks (ANNs) such as the multilayer perceptron (MLP) network and radial basis function (RBF) network have been well documented. The solutions to these problems have inspired a considerable amount of research, one particular area being the application of evolutionary search algorithms such as the genetic algorithm (GA). To date, the vast majority of GA solutions have been aimed at the MLP network. This paper begins with a brief overview of feedforward ANNs and GAs followed by a review of the current state of research in applying evolutionary techniques to training RBF networks.     

基于自适应多目标进化CNN的图像分割方法

卷积神经网络已经成为强大的分割模型,但通常为手动设计,这需要大量时间并且可能导致庞大而复杂的网络.人们对自动设计能够准确分割特定领域图像的高效网络架构越来越感兴趣,然而大部分方法或者没有考虑构建更加灵活的网络架构,或者没有考虑多个目标优化模型.鉴于此,提出一种称为AdaMo-ECNAS的自适应多目标进化卷积神经架构搜索算法,用于特定领域的图像分割,在进化过程中考虑多个性能指标并通过优化模型的多目标适应特定的数据集. AdaMo-ECNAS可以构建灵活多变的预测分割模型,其网络架构和超参数通过基于多目标进化的算法找到,算法基于自适应PBI实现3个目标进化问题,即提升预测分割的F₁-score、最大限度减少计算成本以及最大限度挖掘额外训练潜能.将AdaMo-ECNAS在两个真实数据集上进行评估,结果表明所提出方法与其他先进算法相比具有较高的竞争性,甚至是超越的.     

Optimization of neural networks through grammatical evolution and a genetic algorithm

This paper proposes a hybrid neuro-evolutive algorithm (NEA) that uses a compact indirect encoding scheme (IES) for representing its genotypes (a set of ten production rules of a Lindenmayer System with memory), moreover has the ability to reuse the genotypes and automatically build modular, hierarchical and recurrent neural networks. A genetic algorithm (GA) evolves a Lindenmayer System (L-System) that is used to design the neural network’s architecture. This basic neural codification confers scalability and search space reduction in relation to other methods. Furthermore, the system uses a parallel genome scan engine that increases both the implicit parallelism and convergence of the GA. The fitness function of the NEA rewards economical artificial neural networks (ANNs) that are easily implemented. The NEA was tested on five real-world classification datasets and three well-known datasets for time series forecasting (TSF). The results are statistically compared against established state-of-the-art algorithms and various forecasting methods (ADANN, ARIMA, UCM, and Forecast Pro). In most cases, our NEA outperformed the other methods, delivering the most accurate classification and time series forecasting with the least computational effort. These superior results are attributed to the improved effectiveness and efficiency of NEA in the decision-making process. The result is an optimized neural network architecture for solving classification problems and simulating dynamical systems.     

Evolutionary optimization of sparsely connected and time-lagged neural networks for time series forecasting

Time series forecasting (TSF) is an important tool to support decision making (e.g., planning production resources). Artificial neural networks (ANNs) are innate candidates for TSF due to advantages such as nonlinear learning and noise tolerance. However, the search for the best model is a complex task that highly affects the forecasting performance. In this work, we propose two novel evolutionary artificial neural networks (EANNs) approaches for TSF based on an estimation distribution algorithm (EDA) search engine. The first new approach consist of sparsely connected evolutionary ANN (SEANN), which evolves more flexible ANN structures to perform multi-step ahead forecasts. The second one, consists of an automatic Time lag feature selection EANN (TEANN) approach that evolves not only ANN parameters (e.g., input and hidden nodes, training parameters) but also which set of time lags are fed into the forecasting model. Several experiments were held, using a set of six time series, from different real-world domains. Also, two error metrics (i.e., mean squared error and symmetric mean absolute percentage error) were analyzed. The two EANN approaches were compared against a base EANN (with no ANN structure or time lag optimization) and four other methods (autoregressive integrated moving average method, random forest, echo state network and support vector machine). Overall, the proposed SEANN and TEANN methods obtained the best forecasting results. Moreover, they favor simpler neural network models, thus requiring less computational effort when compared with the base EANN.     

Optimized scenario for rainfall forecasting using genetic algorithm coupled with artificial neural network

Rainfall forecasting plays many important role in water resources studies such as river training works and design of flood warning systems. Recent advancement in artificial intelligence and in particular techniques aimed at converting input to output for highly nonlinear, non-convex and dimensionalized processes such as rainfall field, provide an alternative approach for developing rainfall forecasting model. Artificial neural networks (ANNs), which perform a nonlinear mapping between inputs and outputs, are such a technique. Current literatures on artificial neural networks show that the selection of network architecture and its efficient training procedure are major obstacles for their daily usage. In this paper, feed-forward type networks will be developed to simulate the rainfall field and a so-called back propagation (BP) algorithm coupled with genetic algorithm (GA) will be used to train and optimize the networks. The technique will be implemented to forecast rainfall for a number of times using rainfall hyetograph of recording rain gauges in the Upper Parramatta catchment in the western suburbs of Sydney, Australia. Results of the study showed the structuring of ANN network with the input parameter selection, when coupled with GA, performed better compared to similar work of using ANN alone.     

轴承故障融合诊断方法

针对传统人工神经网络在故障诊断中应用的局限性 ,提出一种基于小波变换、遗传算法与神经网络的融合故障诊断方法。该方法先用小波变换对原始采样信号进行特征提取 ,再用遗传算法优化选择最为重要的特征作为神经网络的输入参数。最后 ,由神经网络进行状态识别和特征分类。这样不仅减少网络训练时间 ,降低网络计算量 ,而且有效提高分类的准确性及故障诊断的可靠性。轴承故障诊断实验结果表明 ,该方法是有效的。     

Fractional-Neuro-Optimizer: A Neural-Network-Based Optimization Method

Up to now, there have been many attempts in the use of artificial neural networks (ANNs) for solving optimization problems and some types of ANNs, such as Hopfield network and Boltzmann machine, have been applied for combinatorial optimization problems. However, there are some restrictions in the use of ANNs as optimizers. For example: (1) ANNs cannot optimize continuous variable problems; (2) discrete problems should be mapped into the neural networks’ architecture; and (3) most of the existing neural networks are applicable only for a class of smooth optimization problems and global convexity conditions on the objective functions and constraints are required. In this paper, we introduce a new procedure for stochastic optimization by a recurrent ANN. The concept of fractional calculus is adopted to propose a novel weight updating rule. The introduced method is called fractional-neuro-optimizer (FNO). This method starts with an initial solution and adjusts the network’s weights by a new heuristic and unsupervised rule to reach a good solution. The efficiency of FNO is compared to the genetic algorithm and particle swarm optimization techniques. Finally, the proposed FNO is used for determining the parameters of a proportional–integral–derivative controller for an automatic voltage regulator power system and is applied for designing the water distribution networks.     

AN ANALYSIS OF NEURAL NETWORKS AS SIMULATORS AND EMULATORS

Recent advances in artificial neural networks (ANNs) have led to the design and construction of neuroarchitectures as simulator and emulators of a variety of problems in science and engineering. Such problems include pattern recognition, prediction, optimization, associative memory, and control of dynamic systems. This paper offers an analytical overview of the most successful design, implementation, and application of neuroarchitectures as neurosimulators and neuroemulators. It also outlines historical notes on the formulation of basic biological neuron, artificial computational models, network architectures, and learning processes of the most common ANN; describes and analyzes neurosimulation on parallel architecture both in software and hardware (neurohardware); presents the simulation of ANNs on parallel architectures; gives a brief introduction of ANNs in vector microprocessor systems; and presents ANNs in terms of the "new technologies". Specifically, it discusses cellular computing, cellular neural networks (CNNs), a new proposition for unsupervised neural networks (UNNs), and pulse coupled neural networks (PCNNs).     

Model selection of extreme learning machine based on multi-objective optimization

As a novel learning algorithm for single-hidden-layer feedforward neural networks, extreme learning machines (ELMs) have been a promising tool for regression and classification applications. However, it is not trivial for ELMs to find the proper number of hidden neurons due to the nonoptimal input weights and hidden biases. In this paper, a new model selection method of ELM based on multi-objective optimization is proposed to obtain compact networks with good generalization ability. First, a new leave-one-out (LOO) error bound of ELM is derived, and it can be calculated with negligible computational cost once the ELM training is finished. Furthermore, the hidden nodes are added to the network one-by-one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this LOO bound and the norm of output weight simultaneously in order to avoid over-fitting. Experiments on five UCI regression data sets are conducted, demonstrating that the proposed algorithm can generally obtain better generalization performance with more compact network than the conventional gradient-based back-propagation method, original ELM and evolutionary ELM.     

基于粗糙集理论的神经网络研究及应用

为了补偿神经网络的黑箱特性并提高其工作性能,将粗糙集理论同神经网络结合起来,提出一种基于粗糙集的神经网络体系结构．首先,利用粗糙集理论对神经网络初始化参数的选择和确定进行指导,赋予各参数相关的物理意义;然后,以系统输出误差最小化为目标对粗糙神经网络进行训练,使其满足性能要求．实验结果表明,粗糙神经网络能较好地完成数据挖掘任务,并能获得较高的分类精度．     

A parallel implementation of the backward error propagation neural network training algorithm: experiments in event identification.

An artificial neural-network-based (ANN) event detection and alarm generation system has been developed to aid clinicians in the identification of critical events commonly occurring in the anesthesia breathing circuit. To detect breathing circuit problems, the system monitored CO2 gas concentration, gas flow, and airway pressure. Various parameters were extracted from each of these input waveforms and fed into an artificial neural network. To develop truly robust ANNs, investigators are required to train their networks on large training data sets, requiring enormous computing power. We implemented a parallel version of the backward error propagation neural network training algorithm in the widely portable parallel programming language C-Linda. A maximum speedup of 4.06 was obtained with six processors. This speedup represents a reduction in total run-time from 6.4 to 1.5 h. By reducing the total run time of the computation through parallelism, we were able to optimize many of the neural network's initial parameters. We conclude that use of the master-worker model of parallel computation is an excellent method for speeding up the backward error propagation neural network training algorithm.     

Memetic evolutionary training for recurrent neural networks: an application to time-series prediction

Abstract: Artificial neural networks are bio-inspired mathematical models that have been widely used to solve complex problems. The training of a neural network is an important issue to deal with, since traditional gradient-based algorithms become easily trapped in local optimal solutions, therefore increasing the time taken in the experimental step. This problem is greater in recurrent neural networks, where the gradient propagation across the recurrence makes the training difficult for long-term dependences. On the other hand, evolutionary algorithms are search and optimization techniques which have been proved to solve many problems effectively. In the case of recurrent neural networks, the training using evolutionary algorithms has provided promising results. In this work, we propose two hybrid evolutionary algorithms as an alternative to improve the training of dynamic recurrent neural networks. The experimental section makes a comparative study of the algorithms proposed, to train Elman recurrent neural networks in time-series prediction problems.     

Analysis of learning rate and momentum term in backpropagation neural network algorithm trained to predict pavement performance

Pavement performance modeling is a critical component of any pavement management system (PMS) decision-making process. A characteristic feature of pavement performance models is that they are formulated and estimated statistically from field data. The statistical modeling can only consider no more than a few of the parameters, in a simplified manner, and in some cases various transformations of the original data. Lately, artificial neural networks (ANNs) were applied to pavement performance modeling. The ANNs offer a number of advantages over the traditional statistical methods, caused by their generalization, massive parallelism and ability to offer real time solutions. Unfortunately, in pavement performance modeling, only simulated data were used in ANNs environment. In this paper, real pavement condition and traffic data and specific architecture are used to investigate the effect of learning rate and momentum term on back-propagation algorithm neural network trained to predict flexible pavement performance. On the basis of the analysis it is concluded that an extremely low learning rate around 0.001–0.005 combination and momentum term between 0.5–0.9 do not give satisfactory results for the specific data set and the architecture used. It is also established that the learning rate and momentum term, and validation data can be used to identify when over-learning is taking place in a training set.     

Induction motors bearing fault detection using pattern recognition techniques

This paper proposes a systematic procedure based on a pattern recognition technique for fault diagnosis of induction motors bearings through the artificial neural networks (ANNs). In this method, the use of time domain features as a proper alternative to frequency features is proposed to improve diagnosis ability. The features are obtained from direct processing of the signal segments using very simple calculation. Three different cases including, healthy, inner race defect and outer race defect are investigated using the proposed algorithm. The ANNs are trained with a subset of the experimental data for known machine conditions. Once the network is trained, efficiency of the proposed method is evaluated using the remaining set of data. The obtained results indicate that using time domain features can be effective in accurate diagnosis of various motor bearing faults with high precision and low computational burden.     

Estimation of heat transfer in oscillating annular flow using artifical neural networks

In this study, the prediction of heat transfer from a surface having constant heat flux subjected to oscillating annular flow is investigated using artificial neural networks (ANNs). An experimental study is carried out to estimate the heat transfer characteristics as a function of some input parameters, namely frequency, amplitude, heat flux and filling heights. In the experiments, a piston cylinder mechanism is used to generate an oscillating flow in a liquid column at certain frequency and amplitude. The cycle-averaged values are considered in the calculation of heat transfer using the control volume approach. An experimentally evaluated data set is prepared to be processed with the use of neural networks. Back propagation algorithm, the most common learning method for ANNs, is used for training and testing the network. Results of the experiments and the ANN are in close agreements with errors less than 5%. The study showed that the ANNs could be used effectively for modeling oscillating flow heat transfer in a vertical annular duct.     

基于进化神经网络的短期电网负荷预测算法

本文提出了一种基于进化神经网络的短期电网负荷预测算法。该算法使用了改进的人工蜂群算法与BP神经网络融合生成的进化神经网络,并且使用改进的人工蜂群算法对进化神经网络的偏置和权重进行优化。该算法将火电历史负荷数据作为输入,使用进化神经网络训练预测模型,预测未来一段时间内的电网负荷。首先获取历史负荷数据,然后将收集到的数据应用于进化神经网络模型训练。人工蜂群算法作为一种全局搜索算法,可以有效地探索模型参数空间,寻找最佳的模型参数组合以提升预测精度。为了验证所提出的负荷预测方法的有效性,本文使用了火电网负荷数据进行测试。实验结果表明,在短期电网负荷预测方面,本文提出的进化神经网络比传统方法预测结果更加准确可靠。     

Particle Swarm Optimization of MLP for the identification of factors related to Common Mental Disorders

Social class differences in the prevalence of Common Mental Disorder (CMD) are likely to vary according to time, culture and stage of economic development. The present study aimed to investigate the use of optimization of architecture and weights of Artificial Neural Network (ANN) for identification of the factors related to CMDs. The identification of the factors was possible by optimizing the architecture and weights of the network. The optimization of architecture and weights of ANNs is based on Particle Swarm Optimization with early stopping criteria. This approach achieved a good generalization control, as well as similar or better results than other techniques, but with a lower computational cost, with the ability to generate small networks and with the advantage of the automated architecture selection, which simplify the training process. This paper presents the results obtained in the experiments with ANNs in which it was observed an average percentage of correct classification of individuals with positive diagnostic for the CMDs of 90.59%.     

A hybrid linear/nonlinear training algorithm for feedforward neuralnetworks

This paper presents a new hybrid optimization strategy for training feedforward neural networks. The algorithm combines gradient-based optimization of nonlinear weights with singular value decomposition (SVD) computation of linear weights in one integrated routine. It is described for the multilayer perceptron (MLP) and radial basis function (RBF) networks and then extended to the local model network (LMN), a new feedforward structure in which a global nonlinear model is constructed from a set of locally valid submodels. Simulation results are presented demonstrating the superiority of the new hybrid training scheme compared to second-order gradient methods. It is particularly effective for the LMN architecture where the linear to nonlinear parameter ratio is large.     

Bio-inspired and gradient-based algorithms to train MLPs: The influence of diversity

This paper has three main goals: (i) to employ two classes of algorithms: bio-inspired and gradient-based to train multi-layer perceptron (MLP) neural networks for pattern classification; (ii) to combine the trained neural networks into ensembles of classifiers; and (iii) to investigate the influence of diversity in the classification performance of individual and ensembles of classifiers. The optimization version of an artificial immune network, named opt-aiNet, particle swarm optimization (PSO) and an evolutionary algorithm (EA) are used as bio-inspired methods to train MLP networks. Besides, the standard backpropagation with momentum (BPM), a quasi-Newton method called DFP and a modified scaled-conjugate gradient (SCGM) are the gradient-based algorithms used to train MLP networks in this work. Comparisons among all the training methods are presented in terms of classification accuracy and diversity of the solutions found. The results obtained suggest that most bio-inspired algorithms deteriorate the diversity of solutions during the search, while immune-based methods, like opt-aiNet, and multiple initializations of standard gradient-based algorithms provide diverse solutions that result in good classification accuracy for the ensembles.