循环神经网络研究综述

循环神经网络是神经网络序列模型的主要实现形式,近几年得到迅速发展,其是机器翻译、机器问题回答、序列视频分析的标准处理手段,也是对于手写体自动合成、语音处理和图像生成等问题的主流建模手段.鉴于此,循环神经网络的各分支按照网络结构进行详细分类,大致分为3大类:一是衍生循环神经网络,这类网络是基于基本RNNs模型的结构衍生变体,即对RNNs的内部结构进行修改;二是组合循环神经网络,这类网络将其他一些经典的网络模型或结构与第一类衍生循环神经网络进行组合,得到更好的模型效果,是一种非常有效的手段;三是混合循环神经网络,这类网络模型既有不同网络模型的组合,又在RNNs内部结构上进行修改,是同属于前两类网络分类的结构.为了更加深入地理解循环神经网络,进一步介绍与循环神经网络经常混为一谈的递归神经网络结构以及递归神经网络与循环神经网络的区别和联系.在详略描述上述模型的应用背景、网络结构以及模型变种后,对各个模型的特点进行总结和比较,并对循环神经网络模型进行展望和总结.     

基于周期性建模的时间序列预测方法及电价预测研究

时间序列数据广泛存在于人类的生产生活中, 通常具有复杂的非线性动态和一定的周期性. 与传统的时间序列分析方法相比, 基于深度学习的方法更能捕捉数据的深层特性, 对具有复杂非线性的时间序列有较好的建模效果. 为了在神经网络中显式地建模时间序列数据的周期性和趋势性, 本文在循环神经网络的基础上引入了周期损失和趋势损失, 建立了基于周期性建模和多任务学习的时间序列预测模型. 将模型应用到欧洲能源交易所法国市场的能源市场价格预测中, 结果表明周期损失和趋势损失能够提高神经网络的泛化能力, 并提高预测时间序列趋势的精度.     

Forecasting foreign exchange rates using recurrent neural networks

This article proposes the use of recurrent neural networks in order to forecast foreign exchange rates. Artificial neural networks have proven to be efficient and profitable in fore casting financial time series. In particular, recurrent networks in which activity patterns pass through the network more than once before they generate an output pattern can learn ex tremely complex temporal sequences. Three recurrent architectures are compared in terms of prediction accuracy of futures forecast for Deutsche mark currency. A trading strategy is then devised and optimized. The profitability of the trading strategy taking into account trans action costs is shown for the different architectures. The methods described here which have obtained promising results in real time trading are applicable to other markets.     

Efficient prediction of exchange rates with low complexity artificial neural network models

In recent years forecasting of financial data such as interest rate, exchange rate, stock market and bankruptcy has been observed to be a potential field of research due to its importance in financial and managerial decision making. Survey of existing literature reveals that there is a need to develop efficient forecasting models involving less computational load and fast forecasting capability. The present paper aims to fulfill this objective by developing two novel ANN models involving nonlinear inputs and simple ANN structure with one or two neurons. These are: functional link artificial neural network (FLANN) and cascaded functional link artificial neural network (CFLANN). These have been employed to predict currency exchange rate between US$ to British Pound, Indian Rupees and Japanese Yen. The performance of the proposed models have been evaluated through simulation and have been compared with those obtained from standard LMS based forecasting model. It is observed that the CFLANN model performs the best followed by the FLANN and the LMS models.     

Stability analysis of discrete-time recurrent neural networks based on standard neural network models

In order to conveniently analyze the stability of various discrete-time recurrent neural networks (RNNs), including bidirectional associative memory, Hopfield, cellular neural network, Cohen-Grossberg neural network, and recurrent multiplayer perceptrons, etc., the novel neural network model, named standard neural network model (SNNM) is advanced to describe this class of discrete-time RNNs. The SNNM is the interconnection of a linear dynamic system and a bounded static nonlinear operator. By combining Lyapunov functional with S-Procedure, some useful criteria of global asymptotic stability for the discrete-time SNNMs are derived, whose conditions are formulated as linear matrix inequalities. Most delayed (or non-delayed) RNNs can be transformed into the SNNMs to be stability analyzed in a unified way. Some application examples of the SNNMs to the stability analysis of the discrete-time RNNs shows that the SNNMs make the stability conditions of the RNNs easily verified.     

Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling

This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg–Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.      

Multiobjective hybrid optimization and training of recurrent neural networks.

The application of neural networks to solve a problem involves tasks with a high computational cost until a suitable network is found, and these tasks mainly involve the selection of the network topology and the training step. We usually select the network structure by means of a trial-and-error procedure, and we then train the network. In the case of recurrent neural networks (RNNs), the lack of suitable training algorithms sometimes hampers these procedures due to vanishing gradient problems. This paper addresses the simultaneous training and topology optimization of RNNs using multiobjective hybrid procedures. The proposal is based on the SPEA2 and NSGA2 algorithms for making hybrid methods using the Baldwinian hybridization strategy. We also study the effects of the selection of the objectives, crossover, and mutation in the diversity during evolution. The proposals are tested in the experimental section to train and optimize the networks in the competition on artificial time-series (CATS) benchmark.     

Markovian architectural bias of recurrent neural networks

In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure.     

A combined Independent Component Analysis–Neural Network model for forecasting exchange rate variation

The currency market is one of the most efficient markets, making it very difficult to predict future prices. Several studies have sought to develop more accurate models to predict the future exchange rate by analyzing econometric models, developing artificial intelligence models and combining both through the creation of hybrid models. This paper proposes a hybrid model for forecasting the variations of five exchange rates related to the US Dollar: Euro, British Pound, Japanese Yen, Swiss Franc and Canadian Dollar. The proposed model uses Independent Component Analysis (ICA) to deconstruct the series into independent components as well as neural networks (NN) to predict each component. This method differentiates this study from previous works where ICA has been used to extract the noise of time series or used to obtain explanatory variables that are then used in forecasting. The proposed model is then compared to random walk, autoregressive and conditional variance models, neural networks, recurrent neural networks and long–short term memory neural networks. The hypothesis of this study supposes that first deconstructing the exchange rate series and then predicting it separately would produce better forecasts than traditional models. By using the mean squared error and mean absolute percentage error as a measures of performance and Model Confidence Sets to statistically test the superiority of the proposed model, our results showed that this model outperformed the other models examined and significantly improved the accuracy of forecasts. These findings support this model’s use in future research and in decision-making related to investments.     

Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference

Financial forecasting is an example of a signal processing problem which is challenging due to small sample sizes, high noise, non-stationarity, and non-linearity. Neural networks have been very successful in a number of signal processing applications. We discuss fundamental limitations and inherent difficulties when using neural networks for the processing of high noise, small sample size signals. We introduce a new intelligent signal processing method which addresses the difficulties. The method proposed uses conversion into a symbolic representation with a self-organizing map, and grammatical inference with recurrent neural networks. We apply the method to the prediction of daily foreign exchange rates, addressing difficulties with non-stationarity, overfitting, and unequal a priori class probabilities, and we find significant predictability in comprehensive experiments covering 5 different foreign exchange rates. The method correctly predicts the directionof change for the next day with an error rate of 47.1%. The error rate reduces to around 40% when rejecting examples where the system has low confidence in its prediction. We show that the symbolic representation aids the extraction of symbolic knowledge from the trained recurrent neural networks in the form of deterministic finite state automata. These automata explain the operation of the system and are often relatively simple. Automata rules related to well known behavior such as tr end following and mean reversal are extracted.     

Combining empirical mode decomposition and deep recurrent neural networks for predictive maintenance of lithium-ion battery

Predictive maintenance of lithium-ion batteries has been one of the popular research subjects in recent years. Lithium-ion batteries can be used as the energy supply for industrial equipment, such as automated guided vehicles and battery electric vehicles. Predictive maintenance plays an important role in the application of smart manufacturing. This mechanism can provide different levels of pre-diagnosis for machines or components. Remaining useful life (RUL) prediction is crucial for the implementation of predictive maintenance strategies. RUL refers to the estimated useful life remaining before the machine cannot operate after a certain period of operation. This study develops a hybrid data science model based on empirical mode decomposition (EMD), grey relational analysis (GRA), and deep recurrent neural networks (RNN) for the RUL prediction of lithium-ion batteries. The EMD and GRA methods are first adopted to extract the characteristics of time series data. Then, various deep RNNs, including vanilla RNN, gated recurrent unit, long short-term memory network (LSTM), and bidirectional LSTM, are established to forecast state of health (SOH) and the RUL of lithium-ion batteries. Bayesian optimization is also used to find the best hyperparameters of deep RNNs. Experimental results with the lithium-ion batteries data of NASA Ames Prognostics Data Repository show that the proposed hybrid data science model can accurately predict the SOH and RUL of lithium-ion batteries. The LSTM network has the optimal results. The proposed hybrid data science model with multiple artificial intelligence-based technologies also demonstrates critical digital-technology enablers for digital transformation of smart manufacturing and transportation.     

A Genetic Programming Approach for EUR/USD Exchange Rate Forecasting and Trading

The purpose of this article is to present a novel genetic programming trading technique in the task of forecasting the next day returns when trading the EUR/USD exchange rate based on the exchange rates of historical data. Aiming at testing its effectiveness, we benchmark the forecasting performance of our genetic programming implementation with three traditional strategies (naive strategy, MACD, and a buy & hold strategy) plus a hybrid evolutionary artificial neural network approach. The proposed genetic programming technique was found to demonstrate the highest trading performance in terms of annualized return and information ratio when compared to all other strategies which have been used. When more elaborate trading techniques, such as leverage, were combined with the examined models, the genetic programming approach still presented the highest trading performance. To the best of our knowledge, this is the first time that genetic programming is applied in the problem of effectively modeling and trading with the EUR/USD exchange rate. Our application now offers practitioners with an effective and extremely promising set of results when forecasting in the foreign exchange market. The developed genetic programming environment is implemented using the C++ programming language and includes a variation of the genetic programming algorithm with tournament selection.     

Forecasting and Prediction Applications in the Field of Power Engineering

Within the field of power engineering, forecasting and prediction techniques underpin a number of applications such as fault diagnosis, condition monitoring and planning. These applications can now be enhanced due to the improved forecasting and prediction capabilities offered through the use of artificial neural networks. This paper demonstrates the maturity of neural network based forecasting and prediction through four diverse case studies. In each case study the authors have developed diagnostic, monitoring or planning applications (within the power engineering field) using neural networks and industrial data. The engineering applications discussed in the paper are: condition monitoring and fault diagnosis applied to a power transformer; condition monitoring and fault diagnosis applied to an industrial gas turbine; electrical load forecasting; monitoring of the refuelling process within a nuclear power station. For each case study the data sources, data preparation, neural network methods and implementation of the resulting application is discussed. The paper will show that the forecasting and prediction techniques discussed offer significant engineering benefits in terms of enhanced decision support capabilities.     

Combining seasonal time series ARIMA method and neural networks with genetic algorithms for predicting the production value of the mechanical industry in Taiwan

Supplying industrial firms with an accurate method of forecasting the production value of the mechanical industry to facilitate decision makers in precise planning is highly desirable. Numerous methods, including the autoregressive integrated-moving average (ARIMA) model and artificial neural networks can make accurate forecasts based on historical data. The seasonal ARIMA (SARIMA) model and artificial neural networks can also handle data involving trends and seasonality. Although neural networks can make predictions, deciding the most appropriate input data, network structure and learning parameters are difficult. Therefore, this article presents a hybrid forecasting method that combines the SARIMA model and neural networks with genetic algorithms. Analytical results generated by the SARIMA model are inputted as the input data of a neural network. Subsequently, the number of neurons in the hidden layer and the number of learning parameters of the neural network architecture are globally optimized using genetic algorithms. This model is subsequently adopted to forecast seasonal time series data of the production value of the mechanical industry in Taiwan. The results presented here provide a valuable reference for decision makers in industry.     

A distance-based fuzzy time series model for exchange rates forecasting

Fuzzy time series model has been successfully employed in predicting stock prices and foreign exchange rates. In this paper, we propose a new fuzzy time series model termed as distance-based fuzzy time series (DBFTS) to predict the exchange rate. Unlike the existing fuzzy time series models which require exact match of the fuzzy logic relationships (FLRs), the distance-based fuzzy time series model uses the distance between two FLRs in selecting prediction rules. To predict the exchange rate, a two factors distance-based fuzzy time series model is constructed. The first factor of the model is the exchange rate itself and the second factor comprises many candidate variables affecting the fluctuation of exchange rates. Using the exchange rate data released by the Central Bank of Taiwan, we conducted several experiments on exchange rate forecasting. The experiment results showed that the distance-based fuzzy time series outperformed the random walk model and the artificial neural network model in terms of mean square error.     

A comparison between Fama and French's model and artificial neural networks in predicting the Chinese stock market

Evidence exists that emerging market stock returns are influenced by a different set of factors than those that influence the returns for stocks traded in developed countries. This study uses artificial neural networks to predict stock price movement (i.e., price returns) for firms traded on the Shanghai stock exchange. We compare the predictive power using linear models from financial forecasting literature to the predictive power of the univariate and multivariate neural network models. Our results show that neural networks outperform the linear models compared. These results are statistically significant across our sample firms, and indicate neural networks are a useful tool for stock price prediction in emerging markets, like China.     

Segmented-Memory Recurrent Neural Networks

Conventional recurrent neural networks (RNNs) have difficulties in learning long-term dependencies. To tackle this problem, we propose an architecture called segmented-memory recurrent neural network (SMRNN). A symbolic sequence is broken into segments and then presented as inputs to the SMRNN one symbol per cycle. The SMRNN uses separate internal states to store symbol-level context, as well as segment-level context. The symbol-level context is updated for each symbol presented for input. The segment-level context is updated after each segment. The SMRNN is trained using an extended real-time recurrent learning algorithm. We test the performance of SMRNN on the information latching problem, the “two-sequence problem” and the problem of protein secondary structure (PSS) prediction. Our implementation results indicate that SMRNN performs better on long-term dependency problems than conventional RNNs. Besides, we also theoretically analyze how the segmented memory of SMRNN helps learning long-term temporal dependencies and study the impact of the segment length.      

A Neural-Network-Based Model for the Dynamic Simulation of the Tire/Suspension System While Traversing Road Irregularities

This paper deals with the simulation of the tire/suspension dynamics by using recurrent neural networks (RNNs). RNNs are derived from the multilayer feedforward neural networks, by adding feedback connections between output and input layers. The optimal network architecture derives from a parametric analysis based on the optimal tradeoff between network accuracy and size. The neural network can be trained with experimental data obtained in the laboratory from simulated road profiles (cleats). The results obtained from the neural network demonstrate good agreement with the experimental results over a wide range of operation conditions. The NN model can be effectively applied as a part of vehicle system model to accurately predict elastic bushings and tire dynamics behavior. Although the neural network model, as a black-box model, does not provide a good insight of the physical behavior of the tire/suspension system, it is a useful tool for assessing vehicle ride and noise, vibration, harshness (NVH) performance due to its good computational efficiency and accuracy.      

Forecasting building energy consumption: Adaptive long-short term memory neural networks driven by genetic algorithm

The real-world building can be regarded as a comprehensive energy engineering system; its actual energy consumption depends on complex affecting factors, including various weather data and time signature. Accurate energy consumption forecasting and effective energy system management play an essential part in improving building energy efficiency. The multi-source weather profile and energy consumption data could enable integrating data-driven models and evolutionary algorithms to achieve higher forecasting accuracy and robustness. The proposed building energy consumption forecasting system consists of three layers: data acquisition and storage layer, data pre-processing layer and data analytics layer. The core part of the data analytics layer is a hybrid genetic algorithm (GA) and long-short term memory (LSTM) neural network model for accurate and robust energy prediction. LSTM neural network is adopted to capture the interrelationship between energy consumption data and time. GA is adopted to select the optimal architecture for LSTM neural networks to improve its forecasting accuracy and robustness. The hyper-parameters for determining LSTM architecture include the number of LSTM layers, number of neurons in each LSTM layer, dropping rate of each LSTM layer and network learning rate. Meanwhile, the effects of historical weather profile and time horizon of past information are also investigated. Two real-life educational buildings are adopted to test the performance of the proposed building energy consumption forecasting system. Experiments reveal that the proposed adaptive LSTM neural network performs better than the existing feedforward neural network and LSTM-based prediction models in accuracy and robustness. It also outperforms those LSTM networks whose hyper-parameters are determined by grid search, Bayesian optimisation and PSO. Such accurate energy consumption prediction can play an essential role in various areas, including daily building energy management, decision making of facility managers, building information model designs, net-zero energy operation, climate change mitigation and circular economy.     

Evolution of cartesian genetic programs for development of learning neural architecture

Although artificial neural networks have taken their inspiration from natural neurological systems, they have largely ignored the genetic basis of neural functions. Indeed, evolutionary approaches have mainly assumed that neural learning is associated with the adjustment of synaptic weights. The goal of this paper is to use evolutionary approaches to find suitable computational functions that are analogous to natural sub-components of biological neurons and demonstrate that intelligent behavior can be produced as a result of this additional biological plausibility. Our model allows neurons, dendrites, and axon branches to grow or die so that synaptic morphology can change and affect information processing while solving a computational problem. The compartmental model of a neuron consists of a collection of seven chromosomes encoding distinct computational functions inside the neuron. Since the equivalent computational functions of neural components are very complex and in some cases unknown, we have used a form of genetic programming known as Cartesian genetic programming (CGP) to obtain these functions. We start with a small random network of soma, dendrites, and neurites that develops during problem solving by repeatedly executing the seven chromosomal programs that have been found by evolution. We have evaluated the learning potential of this system in the context of a well-known single agent learning problem, known as Wumpus World. We also examined the harder problem of learning in a competitive environment for two antagonistic agents, in which both agents are controlled by independent CGP computational networks (CGPCN). Our results show that the agents exhibit interesting learning capabilities.