Multiplicative neuron model artificial neural network based on Gaussian activation function

Multiplicative neuron model-based artificial neural networks are one of the artificial neural network types which have been proposed recently and have produced successful forecasting results. Sigmoid activation function was used in multiplicative neuron model-based artificial neural networks in the previous studies. Although artificial neural networks which involve the use of radial basis activation function produce more successful forecasting results, Gaussian activation function has not been used for multiplicative neuron model yet. In this study, rather than using a sigmoid activation function, Gaussian activation function was used in multiplicative neuron model artificial neural network. The weights of artificial neural network and parameters of activation functions were optimized by guaranteed convergence particle swarm optimization. Two major contributions of this study are as follows: the use of Gaussian activation function in multiplicative neuron model for the first time and the optimizing of central and propagation parameters of activation function with the weights of artificial neural network in a single optimization process. The superior forecasting performance of the proposed Gaussian activation function-based multiplicative neuron model artificial neural network was proved by applying it to real-life time series.     

自适应差分进化算法优化BP神经网络的时间序列预测

针对BP神经网络学习算法随机初始化连接权值和阈值易使模型陷入局部极小点的缺点,设计了一种自适应差分进化算法优化BP神经网络的混合算法。该混合算法中,差分进化算法采用自适应变异和交叉因子优化BP神经网络的初始权值和阈值,再用预寻优得到的初始权值和阈值训练BP神经网络得到最优的权值和阈值。首先对改进的自适应差分进化算法运用测试函数进行性能测试,然后用一个经典时间序列问题对提出的混合算法进行了检验,并与一般的神经网络、ARIMA预测模型及其它混合预测模型进行了对比,实验结果表明,本文提出的混合算法有效并且明显提高了预测精度。     

Support vector machine with adaptive parameters in financial time series forecasting

A novel type of learning machine called support vector machine (SVM) has been receiving increasing interest in areas ranging from its original application in pattern recognition to other applications such as regression estimation due to its remarkable generalization performance. This paper deals with the application of SVM in financial time series forecasting. The feasibility of applying SVM in financial forecasting is first examined by comparing it with the multilayer back-propagation (BP) neural network and the regularized radial basis function (RBF) neural network. The variability in performance of SVM with respect to the free parameters is investigated experimentally. Adaptive parameters are then proposed by incorporating the nonstationarity of financial time series into SVM. Five real futures contracts collated from the Chicago Mercantile Market are used as the data sets. The simulation shows that among the three methods, SVM outperforms the BP neural network in financial forecasting, and there are comparable generalization performance between SVM and the regularized RBF neural network. Furthermore, the free parameters of SVM have a great effect on the generalization performance. SVM with adaptive parameters can both achieve higher generalization performance and use fewer support vectors than the standard SVM in financial forecasting.     

Visualization of Neural Network Predictions for Weather Forecasting

Recurrent neural networks are prime candidates for learning evolutions in multi‐dimensional time series data. The performance of such a network is judged by the loss function, which is aggregated into a scalar value that decreases during training. Observing only this number hides the variation that occurs within the typically large training and testing data sets. Understanding these variations is of highest importance to adjust network hyper‐parameters, such as the number of neurons, number of layers or to adjust the training set to include more representative examples. In this paper, we design a comprehensive and interactive system that allows users to study the output of recurrent neural networks on both the complete training data and testing data. We follow a coarse‐to‐fine strategy, providing overviews of annual, monthly and daily patterns in the time series and directly support a comparison of different hyper‐parameter settings. We applied our method to a recurrent convolutional neural network that was trained and tested on 25 years of climate data to forecast meteorological attributes, such as temperature, pressure and wind velocity. We further visualize the quality of the forecasting models, when applied to various locations on the Earth and we examine the combination of several forecasting models.     

A fuzzy ensemble of parallel polynomial neural networks with information granules formed by fuzzy clustering

In this paper, we introduce a new category of fuzzy models called a fuzzy ensemble of parallel polynomial neural network (FEP²N²), which consist of a series of polynomial neural networks weighted by activation levels of information granules formed with the use of fuzzy clustering. The two underlying design mechanisms of the proposed networks rely on information granules resulting from the use of fuzzy C-means clustering (FCM) and take advantage of polynomial neural networks (PNNs).The resulting model comes in the form of parallel polynomial neural networks. In the design procedure, in order to estimate the optimal values of the coefficients of polynomial neural networks we use a weighted least square estimation algorithm. We incorporate various types of structures as the consequent part of the fuzzy model when using the learning algorithm. Among the diverse structures being available, we consider polynomial neural networks, which exhibit highly nonlinear characteristics when being viewed as local learning models.We use FCM to form information granules and to overcome the high dimensionality problem. We adopt PNNs to find the optimal local models, which can describe the relationship between the input variables and output variable within some local region of the input space.We show that the generalization capabilities as well as the approximation abilities of the proposed model are improved as a result of using polynomial neural networks. The performance of the network is quantified through experimentation in which we use a number of benchmarks already exploited within the realm of fuzzy or neurofuzzy modeling.     

Static, Dynamic, and Hybrid Neural Networks in Forecasting Inflation

The back-propagation neural network (BPN) model has been the most popular form of artificial neural network model used for forecasting, particularly in economics and finance. It is a static (feed-forward) model which has a learning process in both hidden and output layers. In this paper we compare the performance of the BPN model with that of two other neural network models, viz., the radial basis function network (RBFN) model and the recurrent neural network (RNN) model, in the context of forecasting inflation. The RBFN model is a hybrid model with a learning process that is much faster than the BPN model and that is able to generate almost the same results as the BPN model. The RNN model is a dynamic model which allows feedback from other layers to the input layer, enabling it to capture the dynamic behavior of the series. The results of the ANN models are also compared with those of the econometric time series models.     

Comparison of new activation functions in neural network for forecasting financial time series

In artificial neural networks (ANNs), the activation function most used in practice are the logistic sigmoid function and the hyperbolic tangent function. The activation functions used in ANNs have been said to play an important role in the convergence of the learning algorithms. In this paper, we evaluate the use of different activation functions and suggest the use of three new simple functions, complementary log-log, probit and log-log, as activation functions in order to improve the performance of neural networks. Financial time series were used to evaluate the performance of ANNs models using these new activation functions and to compare their performance with some activation functions existing in the literature. This evaluation is performed through two learning algorithms: conjugate gradient backpropagation with Fletcher–Reeves updates and Levenberg–Marquardt.     

基于改进BP网络模型的公路流量预测

针对公路客货运量预测的问题,对现有的常用预测方法进行研究,提出改进BP神经网络预测模型。该模型首先采用动态陡度因子改变激励函数的陡峭程度,改善激励函数的响应特征,得到更好的非线性表达能力;其次利用附加动量因子,通过将以前的经验进行积累,降低了神经网络对误差曲面的局部细节敏感性,较好地遏制网络陷于局部最小;再次采取变学习率学习算法,先给一个较大初值,随着学习过程的进行,学习率不断减小,网络趋于稳定。改进BP算法既可以找到更优解,又可以缩短训练时间。结合某地区的公路运量相关数据,对改进BP神经网络预测模型进行了验证。实验结果表明,该模型的相对误差和迭代次数都取得了较大的改善,对公路客货运量预测很有效。     

Application of temporal difference learning rules in short‐term traffic flow prediction

Studying dynamic behaviours of a transportation system requires the use of the system mathematical models as well as prediction of traffic flow in the system. Therefore, traffic flow prediction plays an important role in today's intelligent transportation systems. This article introduces a new approach to short‐term daily traffic flow prediction based on artificial neural networks. Among the family of neural networks, multi‐layer perceptron (MLP), radial basis function (RBF) neural network and wavenets have been selected as the three best candidates for performing traffic flow prediction. Moreover, back‐propagation (BP) has been adapted as the most efficient learning scheme in all the cases. It is shown that the coefficients produced by temporal signals improve the performance of the BP learning (BPL) algorithm. Temporal signals provide researchers with a new model of temporal difference BP learning algorithm (TDBPL). The capability and performance of TDBPL algorithm are examined by means of simulation in order to prove that the wavelet theory, with its multi‐resolution ability in comparison to RBF neural networks, is a suitable algorithm in traffic flow forecasting. It is also concluded that despite MLP applications, RBF neural networks do not provide negative forecasts. In addition, the local minimum problems are inevitable in MLP algorithms, while RBF neural networks and wavenet networks do not encounter them.     

Using neural network for forecasting TXO price under different volatility models

This study applies backpropagation neural network for forecasting TXO price under different volatility models, including historical volatility, implied volatility, deterministic volatility function, GARCH and GM-GARCH models. The sample period runs from 2008 to 2009, and thus contains the global financial crisis stating in October 2008. Besides RMSE, MAE and MAPE, this study introduces the best forecasting performance ratio (BFPR) as a new performance measure for use in option pricing. The analytical result reveals that forecasting performances are related to the moneynesses, volatility models and number of neurons in the hidden layer, but are not significantly related to activation functions. Implied and deterministic volatility function models have the largest and second largest BFPR regardless of moneyness. Particularly, the forecasting performance in 2008 was significantly inferior to that in 2009, demonstrating that the global financial crisis during October 2008 may have strongly influenced option pricing performance.     

SIMO傅里叶三角基神经网络的权值直接确定法和结构自确定算法

根据傅里叶级数逼近理论,将正交三角函数系作为隐层神经元激励函数,合理选取这些激励函数的周期参数,构造单输入多输出(SIMO)傅里叶三角基神经网络模型.根据该网络的特点,推导出一种基于伪逆的权值直接确定法,从而1步计算出网络最优权值,并在此基础上设计出隐层结构自确定算法.仿真结果表明,与传统BP(反向传播)神经网络及基于最小二乘法的SIMO傅里叶神经网络模型相比,本网络模型具有更高的计算精度和更快的计算速度.     

The extreme learning machine learning algorithm with tunable activation function

In this paper, we propose an extreme learning machine (ELM) with tunable activation function (TAF-ELM) learning algorithm, which determines its activation functions dynamically by means of the differential evolution algorithm based on the input data. The main objective is to overcome the problem dependence of fixed slop of the activation function in ELM. We mainly considered the issue of processing of benchmark problems on function approximation and pattern classification. Compared with ELM and E-ELM learning algorithms with the same network size or compact network configuration, the proposed algorithm has improved generalization performance with good accuracy. In addition, the proposed algorithm also has very good performance in the TAF neural networks learning algorithms.     

Bi-modal derivative adaptive activation function sigmoidal feedforward artificial neural networks

In this work an adaptive mechanism for choosing the activation function is proposed and described. Four bi-modal derivative sigmoidal adaptive activation function is used as the activation function at the hidden layer of a single hidden layer sigmoidal feedforward artificial neural networks. These four bi-modal derivative activation functions are grouped as asymmetric and anti-symmetric activation functions (in groups of two each). For the purpose of comparison, the logistic function (an asymmetric function) and the function obtained by subtracting 0.5 from it (an anti-symmetric) function is also used as activation function for the hidden layer nodes’. The resilient backpropagation algorithm with improved weight-tracking (iRprop⁺) is used to adapt the parameter of the activation functions and also the weights and/or biases of the sigmoidal feedforward artificial neural networks. The learning tasks used to demonstrate the efficacy and efficiency of the proposed mechanism are 10 function approximation tasks and four real benchmark problems taken from the UCI machine learning repository. The obtained results demonstrate that both for asymmetric as well as anti-symmetric activation usage, the proposed/used adaptive activation functions are demonstratively as good as if not better than the sigmoidal function without any adaptive parameter when used as activation function of the hidden layer nodes.     

基于复合正交神经网络的自适应逆控制系统

目前，在自适应逆控制系统中常采用BP神经网络，而BP网络存在算法复杂、易陷入局部极小解等不足。而正交神经网络能克服BP网络的不足，但由于正交神经网络学习算法存在某些局限性，提出了一种复合正交神经网络，该正交网络结构与三层前向正交网络相同，不同的是正交网络的隐单元处理函数采用带参数的Sigmoid函数的复合正交函数，该神经网络算法简单，学习收敛速度快，并能对网络的函数参数进行优化，为非线性系统的动态建模提供了一种方法。仿真实验表明，网络在用于过程的自适应逆控制中具有很高的控制精度和自适应学习能力。该动态神经网络比其它神经网络具有更强的建模能力与学习适应性，有线性、非线性逼近精度高等优异特性，非常适合于实时控制系统。     

Local recurrent sigmoidal-wavelet neurons in feed-forward neural network for forecasting of dynamic systems: Theory

In this paper different structure of the neurons in the hidden layer of a feed-forward network, for forecasting of the dynamic systems, are proposed. Each neuron in the network is a combination of the sigmoidal activation function (SAF) and wavelet activation function (WAF). The output of the hidden neuron is the product of the output from these two activation functions. A delay element is used to feedback the output of the sigmoidal and the wavelet activation function to each other. This arrangement leads to proposed five possible configurations of recurrent neurons. Besides proposing these neuron models, the presented paper tries to compare the performance of wavelet function with sigmoid function. To guarantee the stability and the convergence of the learning process, upper bound for the learning rates has been investigated using the Lyapunov stability theorem. A two-phase adaptive learning rate ensures this upper bound. Universal approximation property of the feed-forward network with the proposed neurons has also been investigated. Finally, the applicability and comparison of the proposed recurrent networks has been weathered on two benchmark problem catering different types of dynamical systems.     

Training algorithm for radial basis function neural network based on quantum-behaved particle swarm optimization

Radial basis function (RBF) networks are widely applied in function approximation, system identification, chaotic time series forecasting, etc. To use a RBF network, a training algorithm is absolutely necessary for determining the network parameters. The existing training algorithms, such as orthogonal least squares (OLS) algorithm, clustering and gradient descent algorithm, have their own shortcomings respectively. In this paper, we propose a training algorithm based on a novel population-based evolutionary technique, quantum-behaved particle swarm optimization (QPSO), to train RBF neural network. The proposed QPSO-trained RBF network was tested on non-linear system identification problem and chaotic time series forecasting problem, and the results show that it can identify the system and forecast the chaotic time series more quickly and precisely than that trained by the particle swarm algorithm.     

多分辨小波过程神经网络及其应用研究

以小波多分辨分析为理论基础,结合过程神经网络模型,建立了具有分层、多分辨和局部学习特性的多分辨小波过程神经网络.该网络充分利用小波函数和尺度函数互补的特性,以及过程神经网络可以处理连续输入信号的能力,能够很好地解决复杂的非线性时间序列的预测问题.给出了相应的学习算法,并以航空发动机排气温度裕度状态监视为例,利用多分辨小波过程神经网络进行预测.结果表明,多分辨小波过程神经网络收敛速度快、精度高.同时也为航空发动机排气温度裕度状态监视问题提供了一种有效的方法.     

Immunological Algorithm-based Neural Network Learning for Sales Forecasting

This study proposes a variation immunological system (VIS) algorithm with radial basis function neural network (RBFN) learning for function approximation and the exercise of industrial computer (IC) sales forecasting. The proposed VIS algorithm was applied to the RBFN to execute the learning process for adjusting the network parameters involved. To compare the performance of relevant algorithms, three benchmark problems were used to justify the results of the experiment. With better accuracy in forecasting, the trained RBFN can be practically utilized in the IC sales forecasting exercise to make predictions and could enhance business profit.     

An adiabatic neural network for RBF approximation

Numerous studies have addressed nonlinear functional approximation by multilayer perceptrons (MLPs) and RBF networks as a special case of the more general mapping problem. The performance of both these supervised network models intimately depends on the efficiency of their learning process. This paper presents an unsupervised recurrent neural network, based on the recurrent Mean Field Theory (MFT) network model, that finds a least-squares approximation to an arbitrary L₂ function, given a set of Gaussian radially symmetric basis functions (RBFs). Essential is the reformulation of RBF approximation as a problem of constrained optimisation. A new concept of adiabatic network organisation is introduced. Together with an adaptive mechanism of temperature control this allows the network to build a hierarchical multiresolution approximation with preservation of the global optimisation characteristics. A revised problem mapping results in a position invariant local interconnectivity pattern, which makes the network attractive for electronic implementation. The dynamics and performance of the network are illustrated by numerical simulation.     

Performance of fuzzy logic-based slope tuning of neural equaliser for digital communication channel

Adaptive equalisation in digital communication systems is a process of compensating the disruptive effects caused mainly by intersymbol interference in a band-limited channel and plays a vital role for enabling higher data rate in modern digital communication system. Designing efficient equalisers having low structural complexity and faster learning algorithms is also an area of much research interest in the present scenario. This paper presents a novel technique of improving the performance of conventional multilayer perceptron (MLP)-based decision feedback equaliser (DFE) of reduced structural complexity by adapting the slope of the sigmoidal activation function using fuzzy logic control technique. The adaptation of the slope parameter increases the degrees of freedom in the weight space of the conventional feedforward neural network (CFNN) configuration. Application of this technique provides faster learning with less training samples and significant performance gain. This research work also proposes adaptive channel equalisation techniques on recurrent neural network framework. Exhaustive simulation studies carried out prove that by replacing the conventional sigmoid activation functions in each of the processing nodes of recurrent neural network with multilevel sigmoid activation functions, the bit error rate performance has significantly improved. Further slopes of different levels of the multilevel sigmoid have been adapted using fuzzy logic control concept. Simulation results considering standard channel models show faster learning with less number of training samples and performance level comparable to the their conventional counterparts. Also, there is scope for parallel implementation of slope adaptation technique in real-time implementation, which saves the computational time.