Robust identification of nonlinear complex systems using low complexity ANN and particle swarm optimization technique

The paper introduces a novel method of adaptive robust identification of complex nonlinear dynamic plants including Box Jenkin, Mackey Glass and Sunspot series under the presence of strong outliers in the training samples. The identification model consists of a low complexity single layer functional link artificial neural network (FLANN) in the feed forward path and another on the feedback path. The connecting weights are iteratively adjusted by a population based particle swarm optimization technique so that a robust cost function (RCF) of the model-error is minimized. To demonstrate robust identification performance up to 50% random samples of the plant output is contaminated with strong outliers and are employed for training the model using PSO tool. Identification of wide varieties of benchmark complex static and dynamic plants is carried out through simulation study and the performance obtained are compared with those obtained from using standard squared error norm as CF. It is in general observed that, the Wilcoxon norm provides best identification performance compared to squared error and other RCFs based models.     

Predicting river daily flow using wavelet-artificial neural networks based on regression analyses in comparison with artificial neural networks and support vector machine models

This study investigates the ability of wavelet-artificial neural networks (WANN) for the prediction of short-term daily river flow. The WANN model is improved by conjunction of two methods, discrete wavelet transform and artificial neural networks (ANN) based on regression analyses, respectively. The proposed WANN models are applied to the daily flow data of Vanyar station, on the Ajichai River in the northwest region of Iran, and compared with the ANN and support vector machine (SVM) techniques. Mean square error (MSE), mean absolute error (MAE) and correlation coefficient (R) statistics are used for evaluating precision of the WANN, ANN and SVM models. Comparison results demonstrate that the WANN model performs better than the ANN and SVM models in short-term (1-, 2- and 3-day ahead) daily river flow prediction.

     

Preliminary study on wilcoxon learning machines.

As is well known in statistics, the resulting linear regressors by using the rank-based Wilcoxon approach to linear regression problems are usually robust against (or insensitive to) outliers. This motivates us to introduce in this paper the Wilcoxon approach to the area of machine learning. Specifically, we investigate four new learning machines, namely Wilcoxon neural network (WNN), Wilcoxon generalized radial basis function network (WGRBFN), Wilcoxon fuzzy neural network (WFNN), and kernel-based Wilcoxon regressor (KWR). These provide alternative learning machines when faced with general nonlinear learning problems. Simple weights updating rules based on gradient descent will be derived. Some numerical examples will be provided to compare the robustness against outliers for various learning machines. Simulation results show that the Wilcoxon learning machines proposed in this paper have good robustness against outliers. We firmly believe that the Wilcoxon approach will provide a promising methodology for many machine learning problems.     

Evolving Recurrent Neural Network using Cartesian Genetic Programming to Predict The Trend in Foreign Currency Exchange Rates

Forecasting the foreign exchange rate is an uphill task. Numerous methods have been used over the years to develop an efficient and reliable network for forecasting the foreign exchange rate. This study utilizes recurrent neural networks (RNNs) for forecasting the foreign currency exchange rates. Cartesian genetic programming (CGP) is used for evolving the artificial neural network (ANN) to produce the prediction model. RNNs that are evolved through CGP have shown great promise in time series forecasting. The proposed approach utilizes the trends present in the historical data for its training purpose. Thirteen different currencies along with the trade-weighted index (TWI) and special drawing rights (SDR) is used for the performance analysis of recurrent Cartesian genetic programming-based artificial neural networks (RCGPANN) in comparison with various other prediction models proposed to date. The experimental results show that RCGPANN is not only capable of obtaining an accurate but also a computationally efficient prediction model for the foreign currency exchange rates. The results demonstrated a prediction accuracy of 98.872 percent (using 6 neurons only) for a single-day prediction in advance and, on average, 92% for predicting a 1000 days’ exchange rate in advance based on ten days of data history. The results prove RCGPANN to be the ultimate choice for any time series data prediction, and its capabilities can be explored in a range of other fields.     

The C-loss function for pattern classification

This paper presents a new loss function for neural network classification, inspired by the recently proposed similarity measure called Correntropy. We show that this function essentially behaves like the conventional square loss for samples that are well within the decision boundary and have small errors, and L₀ or counting norm for samples that are outliers or are difficult to classify. Depending on the value of the kernel size parameter, the proposed loss function moves smoothly from convex to non-convex and becomes a close approximation to the misclassification loss (ideal 0–1 loss). We show that the discriminant function obtained by optimizing the proposed loss function in the neighborhood of the ideal 0–1 loss function to train a neural network is immune to overfitting, more robust to outliers, and has consistent and better generalization performance as compared to other commonly used loss functions, even after prolonged training. The results also show that it is a close competitor to the SVM. Since the proposed method is compatible with simple gradient based online learning, it is a practical way of improving the performance of neural network classifiers.     

Flood simulation using parallel genetic algorithm integrated wavelet neural networks

The conventional means of flood simulation and prediction using conceptual hydrological model or artificial neural network (ANN) has provided promising results in recent years. However, it is usually difficult to obtain ideal flood reproducing due to the structure of hydrological model. Back propagation (BP) algorithm of ANN may also reach local optimum when training nodal weights. To improve the mapping capability of neural networks, wavelet function was adopted (WANN) to strengthen the non-linear simulation accuracy and generality. In addition, genetic algorithm is integrated with WANN (GAWANN) to avoid reaching local optimum. Meanwhile, Message Passing Interface (MPI) subroutines are introduced for distributed implement considering the time consumption during nodal weights training. The GAWANN was applied in the flood simulation and prediction in arid area. The test results of 4 independent cases were compared to reveal the relations between historical rainfall and runoff under different time lags. The simulation was also carried out with Xinanjiang model to demonstrate the capability of GAWANN. The numerical experiments in this paper indicated that the parallel GAWANN has strong capability of rain-runoff mapping as well as computational efficiency and is suitable for applications of flood simulation in arid areas.     

On the use of data filtering techniques for credit risk prediction with instance-based models

Many techniques have been proposed for credit risk prediction, from statistical models to artificial intelligence methods. However, very few research efforts have been devoted to deal with the presence of noise and outliers in the training set, which may strongly affect the performance of the prediction model. Accordingly, the aim of the present paper is to systematically investigate whether the application of filtering algorithms leads to an increase in accuracy of instance-based classifiers in the context of credit risk assessment. The experimental results with 20 different algorithms and 8 credit databases show that the filtered sets perform significantly better than the non-preprocessed training sets when using the nearest neighbour decision rule. The experiments also allow to identify which techniques are most robust and accurate when confronted with noisy credit data.     

Prediction of dissolved oxygen in River Calder by noise elimination time series using wavelet transform

Prediction of dissolved oxygen (DO) plays an important role in water resources especially in surface waters such as rivers. The oxygen affects a vast number of other water indicators. In this study, the artificial neural network (ANN) and a hybrid wavelet-ANN (WANN) models were considered to predict thirty minutes dissolved oxygen in the River Calder at the Methley Bridge Station was located in the UK. For the proposed WANN model, the discrete wavelet transform (DWT) was linked to the ANN model for DO prediction. To achieve this aim, the original time series of thirty minutes DO and temperature (T) were decomposed in several sub-time series by DWT, and these new sub-series were imposed to the ANN model. The results were compared with single ANN model. The comparisons were done by some of the widely used relevant physical statistic indices. The Nash–Sutcliffe coefficient values were 0.998 and 0.740 for the WANN and ANN models, respectively. The model computed values of DO by the WANN model were in close agreement with respective measured values in the river water. Elimination noise by DWT model during pre-processing data is one of the abilities of the WANN model to better prediction. Since the results indicate closer approximation of the peak DO values by the WANN model, this model could be used for the simulation of cumulative DO data prediction in thirty minutes ahead.     

小波神经网络在北京河湖水华预测中的应用

针对不同季节水华生长的不同特点,在对水华生长规律研究的基础上,运用小波分析对表征水华的叶绿素信号进行降噪处理,建立一种结合小波变换与神经网络相结合的水华预测模型（WANN模型）,该模型既有神经网络的自学习能力特性,又有小波的局部特性,并将其应用到北京夏季河湖水华预测中。通过小波多分辨率分析,对样本包含的信息进行充分挖掘,提取反映其变化规律的成分,有效避免了原始数据中噪声对网络的干扰,提高网络的性能,WANN模型预测结果与BP网络预测结果对比,具有较高的预测能力,从而获得相对理想的预测效果。     

Robust multilayer neural network based on median neuron model

Multilayer perceptron has been widely used in time series forecasting for last two decades. However, it is a well-known fact that the forecasting performance of multilayer perceptron is negatively affected when data have outliers and this is an important problem. In recent years, some alternative neuron models such as generalized-mean neuron, geometric mean neuron, and single multiplicative neuron have been also proposed in the literature. However, it is expected that forecasting performance of artificial neural network approaches based on these neuron models can be also negatively affected by outliers since the aggregation function employed in these models is based on mean value. In this study, a new multilayer feed forward neural network, which is called median neuron model multilayer feed forward (MNM-MFF) model, is proposed in order to deal with this problem caused by outliers and to reach high accuracy level. In the proposed model, unlike other models suggested in the literature, MNM which has median-based aggregation function is employed. MNM is also firstly defined in this study. MNM-MFF is a robust neural network method since aggregation functions in MNM-MFF are based on median, which is not affected much by outliers. In addition, to train MNM-MFF model, particle swarm optimization method was utilized. MNM-MFF was applied to two well-known time series in order to evaluate the performance of the proposed approach. As a result of the implementation, it was observed that the proposed MNM-MFF model has high forecasting accuracy and it is not affected by outlier as much as multilayer perceptron model. Proposed method brings improvement in 7 % for data without outlier, in 90 % for data with outlier, in 95 % for data with bigger outlier.     

Recurrent neural networks and robust time series prediction

We propose a robust learning algorithm and apply it to recurrent neural networks. This algorithm is based on filtering outliers from the data and then estimating parameters from the filtered data. The filtering removes outliers from both the target function and the inputs of the neural network. The filtering is soft in that some outliers are neither completely rejected nor accepted. To show the need for robust recurrent networks, we compare the predictive ability of least squares estimated recurrent networks on synthetic data and on the Puget Power Electric Demand time series. These investigations result in a class of recurrent neural networks, NARMA(p,q), which show advantages over feedforward neural networks for time series with a moving average component. Conventional least squares methods of fitting NARMA(p,q) neural network models are shown to suffer a lack of robustness towards outliers. This sensitivity to outliers is demonstrated on both the synthetic and real data sets. Filtering the Puget Power Electric Demand time series is shown to automatically remove the outliers due to holidays. Neural networks trained on filtered data are then shown to give better predictions than neural networks trained on unfiltered time series.     

A threshold-varying artificial neural network approach for classification and its application to bankruptcy prediction problem

We propose a threshold-varying artificial neural network (TV-ANN) approach for solving the binary classification problem. Using a set of simulated and real-world data set for bankruptcy prediction, we illustrate that the proposed TV-ANN fares well, both for training and holdout samples, when compared to the traditional backpropagation artificial neural network (ANN) and the statistical linear discriminant analysis. The performance comparisons of TV-ANN with a genetic algorithm-based ANN and a classification tree approach C4.5 resulted in mixed results.     

Efficient prediction of exchange rates with low complexity artificial neural network models

In recent years forecasting of financial data such as interest rate, exchange rate, stock market and bankruptcy has been observed to be a potential field of research due to its importance in financial and managerial decision making. Survey of existing literature reveals that there is a need to develop efficient forecasting models involving less computational load and fast forecasting capability. The present paper aims to fulfill this objective by developing two novel ANN models involving nonlinear inputs and simple ANN structure with one or two neurons. These are: functional link artificial neural network (FLANN) and cascaded functional link artificial neural network (CFLANN). These have been employed to predict currency exchange rate between US$ to British Pound, Indian Rupees and Japanese Yen. The performance of the proposed models have been evaluated through simulation and have been compared with those obtained from standard LMS based forecasting model. It is observed that the CFLANN model performs the best followed by the FLANN and the LMS models.     

Computer-aided prediction of the Al2O3 nanoparticles’ effects on tensile strength and percentage of water absorption of concrete specimens

In the present paper, two models based on artificial neural networks and genetic programming for predicting split tensile strength and percentage of water absorption of concretes containing Al₂O₃ nanoparticles have been developed at different ages of curing. For purpose of building these models, training and testing using experimental results for 144 specimens produced with 16 different mixture proportions were conducted. The data used in the multilayer feed-forward neural networks models and input variables of genetic programming models are arranged in a format of eight input parameters that cover the cement content, nanoparticle content, aggregate type, water content, the amount of superplasticizer, the type of curing medium, Age of curing and number of testing try. According to these input parameters, in the neural networks and genetic programming models, the split tensile strength and percentage of water absorption values of concretes containing Al₂O₃ nanoparticles were predicted. The training and testing results in the neural network and genetic programming models have shown that every two models have strong potential for predicting the split tensile strength and percentage of water absorption values of concretes containing Al₂O₃ nanoparticles. It has been found that NN and GEP models will be valid within the ranges of variables. In neural networks model, as the training and testing ended when minimum error norm of network gained, the best results were obtained, and in genetic programming model, when 4 gens was selected to construct the model, the best results were acquired. Although neural network have predicted better results, genetic programming is able to predict reasonable values with a simpler method rather than neural network.     

Customer churn prediction by hybrid neural networks

As churn management is a major task for companies to retain valuable customers, the ability to predict customer churn is necessary. In literature, neural networks have shown their applicability to churn prediction. On the other hand, hybrid data mining techniques by combining two or more techniques have been proved to provide better performances than many single techniques over a number of different domain problems. This paper considers two hybrid models by combining two different neural network techniques for churn prediction, which are back-propagation artificial neural networks (ANN) and self-organizing maps (SOM). The hybrid models are ANN combined with ANN (ANN + ANN) and SOM combined with ANN (SOM + ANN). In particular, the first technique of the two hybrid models performs the data reduction task by filtering out unrepresentative training data. Then, the outputs as representative data are used to create the prediction model based on the second technique. To evaluate the performance of these models, three different kinds of testing sets are considered. They are the general testing set and two fuzzy testing sets based on the filtered out data by the first technique of the two hybrid models, i.e. ANN and SOM, respectively. The experimental results show that the two hybrid models outperform the single neural network baseline model in terms of prediction accuracy and Types I and II errors over the three kinds of testing sets. In addition, the ANN + ANN hybrid model significantly performs better than the SOM + ANN hybrid model and the ANN baseline model.     

Artificial Neural Network Approach to Data Matrix Laser Direct Part Marking

Certain applications have recently appeared in industry where a traditional bar code printed on a label will not survive because the item to be tracked has to be exposed to harsh environments. Laser direct-part marking is a manufacturing process used to create permanent marks on a substrate that could help to alleviate this problem. In this research, artificial neural networks were employed to model the laser direct-part marking process of Data Matrix symbols on carbon steel substrates. Several experiments were conducted to study the laser direct-part marking process and to generate data to serve as training, validation and testing data sets in the artificial neural networks modeling process. Two performance measures, mean squared error and correlation coefficient, were utilized to assess the performance of the artificial neural network models. Single-output artificial neural network models corresponding to four performance measures specific to the Data Matrix bar code symbology were found to have good learning and predicting capabilities. The single-output artificial neural network models were compared to equivalent multiple linear regression models for validation purposes. The prediction capability of the single-output artificial neural network models with respect to laser direct-part marking of Data Matrix symbols on carbon steel substrates was superior to that of the multiple linear regression models.     

Predicting the performance measures of a message-passing multiprocessor architecture using artificial neural networks

In this paper, we develop multi-layer feed-forward artificial neural network (MFANN) models for predicting the performance measures of a message-passing multiprocessor architecture interconnected by the simultaneous optical multiprocessor exchange bus (SOME-Bus), which is a fiber-optic interconnection network. OPNET Modeler is used to simulate the SOME-Bus multiprocessor architecture and to create the training and testing datasets. The performance of the MFANN prediction models is evaluated using standard error of estimate (SEE) and multiple correlation coefficient (R). Also, the results of the MFANN models are compared with the ones obtained by generalized regression neural network (GRNN), support vector regression (SVR), and multiple linear regression (MLR). It is shown that MFANN models perform better (i.e., lower SEE and higher R) than GRNN-based, SVR-based, and MLR-based models for predicting the performance measures of a message-passing multiprocessor architecture.     

Integration of case-based forecasting, neural network, and discriminant analysis for bankruptcy prediction

Recently, it has been an issue of interest how to integrate classification models to increase the prediction performance. This paper suggests a new structured model with multiple stages. It consists of four phases (training, test, adjustment, and prediction), and three types of input data (training, testing, and generalization). The integrated model is applied for bankruptcy prediction. A statistical model, discriminant analysis and two artificial intelligence models, neural network and case-based forecasting, are used in this study. The integration approach produces higher prediction accuracy than individual models.     

基于特征分布差异的对抗样本检测

诸多神经网络模型已被证明极易遭受对抗样本攻击。对抗样本则是攻击者为模型所恶意构建的输入,通过对原始样本输入添加轻微的扰动,导致其极易被机器学习模型错误分类。这些对抗样本会对日常生活中的高要求和关键应用的安全构成严重威胁,如自动驾驶、监控系统和生物识别验证等应用。研究表明在模型的训练期间,检测对抗样本方式相比通过增强模型来预防对抗样本攻击更为有效,且训练期间神经网络模型的中间隐层可以捕获并抽象样本信息,使对抗样本与干净样本更容易被模型所区分。因此,本文针对神经网络模型中的不同隐藏层,其对抗样本输入和原始自然输入的隐层表示进行统计特征差异进行研究。本文研究表明,统计差异可以在不同层之间进行区别。本文通过确定最有效层识别对抗样本和原始自然训练数据集统计特征之间的差异,并采用异常值检测方法,设计一种基于特征分布的对抗样本检测框架。该框架可以分为广义对抗样本检测方法和条件对抗样本检测方法,前者通过在每个隐层中提取学习到的训练数据表示,得到统计特征后,计算测试集的异常值分数,后者则通过深层神经网络模型对测试数据的预测结果比较,得到对应训练数据的统计特征。本文所计算的统计特征包括到原点的范数距离L2和样本协方差矩阵的顶奇异向量的相关性。实验结果显示了两种检测方法均可以利用隐层信息检测出对抗样本,且对由不同攻击产生的对抗样本均具有较好的检测效果,证明了本文所提的检测框架在检测对抗样本中的有效性。     

The effects of data filtering on neural network learning

This paper describes various experiments involving the pre-processing of training data in order to improve the performance of a neural network classifier. To minimise the effect of noisy training samples we applied k-nearest neighbour filtering to the training data. This effectively detects and eliminates outliers, which would otherwise degrade the learning process of the neural network. For a range of neural network complexities, and for two classification tasks, the filtering operation increased classification accuracy.