A novel criterion to select hidden neuron numbers in improved back propagation networks for wind speed forecasting

This paper analyzes various earlier approaches for selection of hidden neuron numbers in artificial neural networks and proposes a novel criterion to select the hidden neuron numbers in improved back propagation networks for wind speed forecasting application. Either over fitting or under fitting problem is caused because of the random selection of hidden neuron numbers in artificial neural networks. This paper presents the solution of either over fitting or under fitting problems. In order to select the hidden neuron numbers, 151 different criteria are tested by means of the statistical errors. The simulation is performed on collected real-time wind data and simulation results prove that proposed approach reduces the error to a minimal value and enhances forecasting accuracy The perfect building of improved back propagation networks employing the fixation criterion is substantiated based on the convergence theorem. Comparative analyses performed prove the selection of hidden neuron numbers in improved back propagation networks is highly effective in nature.     

Thalassaemia classification by neural networks and genetic programming

This paper presents the use of a neural network and a decision tree, which is evolved by genetic programming (GP), in thalassaemia classification. The aim is to differentiate between thalassaemic patients, persons with thalassaemia trait and normal subjects by inspecting characteristics of red blood cells, reticulocytes and platelets. A structured representation on genetic algorithms for non-linear function fitting or STROGANOFF is the chosen architecture for genetic programming implementation. For comparison, multilayer perceptrons are explored in classification via a neural network. The classification results indicate that the performance of the GP-based decision tree is approximately equal to that of the multilayer perceptron with one hidden layer. But the multilayer perceptron with two hidden layers, which is proven to have the most suitable architecture among networks with different number of hidden layers, outperforms the GP-based decision tree. Nonetheless, the structure of the decision tree reveals that some input features have no effects on the classification performance. The results confirm that the classification accuracy of the multilayer perceptron with two hidden layers can still be maintained after the removal of the redundant input features. Detailed analysis of the classification errors of the multilayer perceptron with two hidden layers, in which a reduced feature set is used as the network input, is also included. The analysis reveals that the classification ambiguity and misclassification among persons with minor thalassaemia trait and normal subjects is the main cause of classification errors. These results suggest that a combination of a multilayer perceptron with a blood cell analysis may give rise to a guideline/hint for further investigation of thalassaemia classification.     

Short-term wind speed forecasting based on a hybrid model

Wind power is currently one of the types of renewable energy with a large generation capacity. However, operation of wind power generation is very challenging because of the intermittent and stochastic nature of the wind speed. Wind speed forecasting is a very important part of wind parks management and the integration of wind power into electricity grids. As an artificial intelligence algorithm, radial basis function neural network (RBFNN) has been successfully applied into solving forecasting problems. In this paper, a novel approach named WTT–SAM–RBFNN for short-term wind speed forecasting is proposed by applying wavelet transform technique (WTT) into hybrid model which hybrids the seasonal adjustment method (SAM) and the RBFNN. Real data sets of wind speed in Northwest China are used to evaluate the forecasting accuracy of the proposed approach. To avoid the randomness caused by the RBFNN model or the RBFNN part of the hybrid model, all simulations in this study are repeated 30 times to get the average. Numerical results show that the WTT–SAM–RBFNN outperforms the persistence method (PM), multilayer perceptron neural network (MLP), RBFNN, hybrid SAM and RBFNN (SAM–RBFNN), and hybrid WTT and RBFNN (WTT–RBFNN). It is concluded that the proposed approach is an effective way to improve the prediction accuracy.     

A constructive design method for two-layer perceptrons and its applicatio to the design of modular neural networks

Abstract: A multilayer perceptron is known to be capable of approximating any smooth function to any desired accuracy if it has a sufficient number of hidden neurons. But its training, based on the gradient method, is usually a time consuming procedure that may converge toward a local minimum, and furthermore its performance is greatly influenced by the number of hidden neurons and their initial weights. Usually these crucial parameters are determined based on the trial and error procedure, requiring much experience on the designer's part.
In this paper, a constructive design method (CDM) has been proposed for a two-layer perceptron that can approximate a class of smooth functions whose feature vector classes are linearly separable. Based on the analysis of a given data set sampled from the target function, feature vectors that can characterize the function'well'are extracted and used to determine the number of hidden neurons and the initial weights of the network. But when the classes of the feature vectors are not linearly separable, the network may not be trained easily, mainly due to the interference among the hyperplanes generated by hidden neurons. Next, to compensate for this interference, a refined version of the modular neural network (MNN) has been proposed where each network module is created by CDM. After the input space has been partitioned into many local regions, a two-layer perceptron constructed by CDM is assigned to each local region. By doing this, the feature vector classes are more likely to become linearly separable in each local region and as a result, the function may be approximated with greatly improved accuracy by MNN. An example simulation illustrates the improvements in learning speed using a smaller number of neurons.     

Clearly defined architectures of neural networks and multilayer perceptron

Neural networks with clearly defined architecture differ in the fact that they make it possible to determine the structure of neural network (number of neurons, layers, connections) on the basis of initial parameters of recognition problem. For these networks, the value of weights determined also analytically. In this paper, we consider the problem of networks with clearly defined architecture transformation into the classical schemes of multilayer perceptron architectures. Such possibility may allow us to combine the advantages of neural networks with clearly defined architecture with the capabilities of multilayer perceptron, that eventually may enable us to speed up and simplify the process of creating and training a neural network.     

Evolutionary optimization of sparsely connected and time-lagged neural networks for time series forecasting

Time series forecasting (TSF) is an important tool to support decision making (e.g., planning production resources). Artificial neural networks (ANNs) are innate candidates for TSF due to advantages such as nonlinear learning and noise tolerance. However, the search for the best model is a complex task that highly affects the forecasting performance. In this work, we propose two novel evolutionary artificial neural networks (EANNs) approaches for TSF based on an estimation distribution algorithm (EDA) search engine. The first new approach consist of sparsely connected evolutionary ANN (SEANN), which evolves more flexible ANN structures to perform multi-step ahead forecasts. The second one, consists of an automatic Time lag feature selection EANN (TEANN) approach that evolves not only ANN parameters (e.g., input and hidden nodes, training parameters) but also which set of time lags are fed into the forecasting model. Several experiments were held, using a set of six time series, from different real-world domains. Also, two error metrics (i.e., mean squared error and symmetric mean absolute percentage error) were analyzed. The two EANN approaches were compared against a base EANN (with no ANN structure or time lag optimization) and four other methods (autoregressive integrated moving average method, random forest, echo state network and support vector machine). Overall, the proposed SEANN and TEANN methods obtained the best forecasting results. Moreover, they favor simpler neural network models, thus requiring less computational effort when compared with the base EANN.     

Centroid based Multilayer Perceptron Networks

In this study we investigate a hybrid neural network architecture for modelling purposes. The proposed network is based on the multilayer perceptron (MLP) network. However, in addition to the usual hidden layers the first hidden layer is selected to be a centroid layer. Each unit in this new layer incorporates a centroid that is located somewhere in the input space. The output of these units is the Euclidean distance between the centroid and the input. The centroid layer clearly resembles the hidden layer of the radial basis function (RBF) networks. Therefore the centroid based multilayer perceptron (CMLP) networks can be regarded as a hybrid of MLP and RBF networks. The presented benchmark experiments show that the proposed hybrid architecture is able to combine the good properties of MLP and RBF networks resulting fast and efficient learning, and compact network structure.     

Forecasting wind speed using empirical mode decomposition and Elman neural network

Because of the chaotic nature and intrinsic complexity of wind speed, it is difficult to describe the moving tendency of wind speed and accurately forecast it. In our study, a novel EMD–ENN approach, a hybrid of empirical mode decomposition (EMD) and Elman neural network (ENN), is proposed to forecast wind speed. First, the original wind speed datasets are decomposed into a collection of intrinsic mode functions (IMFs) and a residue by EMD, yielding relatively stationary sub-series that can be readily modeled by neural networks. Second, both IMF components and residue are applied to establish the corresponding ENN models. Then, each sub-series is predicted using the corresponding ENN. Finally, the prediction values of the original wind speed datasets are calculated by the sum of the forecasting values of every sub-series. Moreover, in the ENN modeling process, the neuron number of the input layer is determined by a partial autocorrelation function. Four prediction cases of wind speed are used to test the performance of the proposed hybrid approach. Compared with the persistent model, back-propagation neural network, and ENN, the simulation results show that the proposed EMD–ENN model consistently has the minimum statistical error of the mean absolute error, mean square error, and mean absolute percentage error. Thus, it is concluded that the proposed approach is suitable for wind speed prediction.     

3-D neural mapper for LDPC sum-product decoder

Present article explores prospects for implementing 3D neural mapper suitable for operation in check nodes of sum-product decoding algorithm. Advantage of such mapper based on neural circuits is the allowed on-chip training, which offers potential for controlled accuracy of the mapping and eventually improved performance in the decoding (due to reduced bit error rate resulting from enhanced accuracy of the operations implemented by trained neural circuits). Simulation results presented in this report show feasibility of employing neural multilayer perceptron with practically acceptable number of hidden neurons to achieve high accuracy of the mapping for decoding based on sum-product algorithm. Specifically 7 hidden neurons are shown to attain accuracy better than 0.1%, which is suitable for implementation in decoding devices, such as low-density parity check (LDPC) decoders. The article is published in the original.     

Time-series prediction with single integrate-and-fire neuron

In this paper, a learning algorithm for a single integrate-and-fire neuron (IFN) is proposed and tested for various applications in which a multilayer perceptron neural network is conventionally used. It is found that a single IFN is sufficient for the applications that require a number of neurons in different hidden layers of a conventional neural network. Several benchmark and real-life problems of classification and time-series prediction have been illustrated. It is observed that the inclusion of some more biological phenomenon in an artificial neural network can make it more powerful.     

A novel model by evolving partially connected neural network for stock price trend forecasting

This paper proposes a novel model by evolving partially connected neural networks (EPCNNs) to predict the stock price trend using technical indicators as inputs. The proposed architecture has provided some new features different from the features of artificial neural networks: (1) connection between neurons is random; (2) there can be more than one hidden layer; (3) evolutionary algorithm is employed to improve the learning algorithm and training weights. In order to improve the expressive ability of neural networks, EPCNN utilizes random connection between neurons and more hidden layers to learn the knowledge stored within the historic time series data. The genetically evolved weights mitigate the well-known limitations of gradient descent algorithm. In addition, the activation function is defined using sin(x) function instead of sigmoid function. Three experiments were conducted which are explained as follows. In the first experiment, we compared the predicted value of the trained EPCNN model with the actual value to evaluate the prediction accuracy of the model. Second experiment studied the over fitting problem which occurred in neural network training by taking different number of neurons and layers. The third experiment compared the performance of the proposed EPCNN model with other models like BPN, TSK fuzzy system, multiple regression analysis and showed that EPCNN can provide a very accurate prediction of the stock price index for most of the data. Therefore, it is a very promising tool in forecasting of the financial time series data.     

Optimized approximation algorithm in neural networks without overfitting.

In this paper, an optimized approximation algorithm (OAA) is proposed to address the overfitting problem in function approximation using neural networks (NNs). The optimized approximation algorithm avoids overfitting by means of a novel and effective stopping criterion based on the estimation of the signal-to-noise-ratio figure (SNRF). Using SNRF, which checks the goodness-of-fit in the approximation, overfitting can be automatically detected from the training error only without use of a separate validation set. The algorithm has been applied to problems of optimizing the number of hidden neurons in a multilayer perceptron (MLP) and optimizing the number of learning epochs in MLP's backpropagation training using both synthetic and benchmark data sets. The OAA algorithm can also be utilized in the optimization of other parameters of NNs. In addition, it can be applied to the problem of function approximation using any kind of basis functions, or to the problem of learning model selection when overfitting needs to be considered.     

Synthetic approach to optimal filtering

As opposed to the analytic approach used in the modern theory of optimal filtering, a synthetic approach is presented. The signal/sensor data, which are generated by either computer simulation or actual experiments, are synthesized into a filter by training a recurrent multilayer perceptron (RMLP) with at least one hidden layer of fully or partially interconnected neurons and with or without output feedbacks. The RMLP, after adequate training, is a recursive filter optimal for the given structure, with the lagged feedbacks carrying the optimal conditional statistics at each time point. Above all, it converges to the minimum variance filter as the number of hidden neurons increases. We call such an RMLP a neural filter. Simulation results show that the neural filters with only a few hidden neurons consistently outperform the extended Kalman filter and even the iterated extended Kalman filter for the simple nonlinear signal/sensor systems considered.     

Entropy-based generation of supervised neural networks for classification of structured patterns

Sperduti and Starita proposed a new type of neural network which consists of generalized recursive neurons for classification of structures. In this paper, we propose an entropy-based approach for constructing such neural networks for classification of acyclic structured patterns. Given a classification problem, the architecture, i.e., the number of hidden layers and the number of neurons in each hidden layer, and all the values of the link weights associated with the corresponding neural network are automatically determined. Experimental results have shown that the networks constructed by our method can have a better performance, with respect to network size, learning speed, or recognition accuracy, than the networks obtained by other methods.     

Lur'e systems with multilayer perceptron and recurrent neuralnetworks: absolute stability and dissipativity

Sufficient conditions for absolute stability and dissipativity of continuous-time recurrent neural networks with two hidden layers are presented. In the autonomous case this is related to a Lur'e system with multilayer perceptron nonlinearity. Such models are obtained after parametrizing general nonlinear models and controllers by a multilayer perceptron with one hidden layer and representing the control scheme in standard plant form. The conditions are expressed as matrix inequalities and can be employed for nonlinear H_∞ control and imposing closed-loop stability in dynamic backpropagation     

Combination of artificial neural-network forecasters for predictionof natural gas consumption

The focus of this paper is on combination of artificial neural-network (ANN) forecasters with application to the prediction of daily natural gas consumption needed by gas utilities. ANN forecasters can model the complex relationship between weather parameters and previous gas consumption with the future consumption. A two-stage system is proposed with the first stage containing two ANN forecasters, a multilayer feedforward ANN and a functional link ANN. These forecasters are initially trained with the error backpropagation algorithm, but an adaptive strategy is employed to adjust their weights during online forecasting. The second stage consists of a combination module to mix the two individual forecasts produced in the first stage. Eight different combination algorithms are examined, they are based on: averaging, recursive least squares, fuzzy logic, feedforward ANN, functional link ANN, temperature space approach, Karmarkar's linear programming algorithm (1984) and adaptive mixture of local experts (modular neural networks). The performance is tested on real data from six different gas utilities. The results indicate that combination strategies based on a single ANN outperform the other approaches.     

MLP neural network models of CMM measuring processes

The objective of this study is to show how a multi-layer perceptron (MLP) neural network can be used to model a CMM measuring process. To date, most MLP-based process models have been established for process mean only. An innovative approach is proposed to model simultaneously the mean and the variation of a CMM process using one integrated MLP architecture. Therefore, the MLP-based model obtained captures not only the process mean but also the process variation information. Selected issues related to neural network training are also discussed. Specifically, the guideline that was proposed by Mirchandani and Cao (1989) for selecting a number of hidden neurons is tested to determine the effects of the number of hidden neurons. The performances of two different learning algorithms - back-propagation with momentum factor (BPM) and the Fletcher-Reeves (FR) algorithm - are studied in terms of CPU time, training error, and generalization error.     

Geometrical interpretation and architecture selection of MLP

A geometrical interpretation of the multilayer perceptron (MLP) is suggested in this paper. Some general guidelines for selecting the architecture of the MLP, i.e., the number of the hidden neurons and the hidden layers, are proposed based upon this interpretation and the controversial issue of whether four-layered MLP is superior to the three-layered MLP is also carefully examined.     

ANNSTLF-a neural-network-based electric load forecasting system

A key component of the daily operation and planning activities of an electric utility is short-term load forecasting, i.e., the prediction of hourly loads (demand) for the next hour to several days out. The accuracy of such forecasts has significant economic impact for the utility. This paper describes a load forecasting system known as ANNSTLF (artificial neural-network short-term load forecaster) which has received wide acceptance by the electric utility industry and presently is being used by 32 utilities across the USA and Canada. ANNSTLF can consider the effect of temperature and relative humidity on the load. Besides its load forecasting engine, ANNSTLF contains forecasters that can generate the hourly temperature and relative humidity forecasts needed by the system. ANNSTLF is based on a multiple ANN strategy that captures various trends in the data. Both the first and the second generation of the load forecasting engine are discussed and compared. The building block of the forecasters is a multilayer perceptron trained with the error backpropagation learning rule. An adaptive scheme is employed to adjust the ANN weights during online forecasting. The forecasting models are site independent and only the number of hidden layer nodes of ANN's need to be adjusted for a new database. The results of testing the system on data from ten different utilities are reported.