Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling

This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg–Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.      

Nonlinear maximum likelihood estimation of electricity spot prices using recurrent neural networks

Electricity spot prices are complex processes characterized by nonlinearity and extreme volatility. Previous work on nonlinear modeling of electricity spot prices has shown encouraging results, and we build on this area by proposing an Expectation Maximization algorithm for maximum likelihood estimation of recurrent neural networks utilizing the Kalman filter and smoother. This involves inference of both parameters and hyper-parameters of the model which takes into account the model uncertainty and noise in the data. The Expectation Maximization algorithm uses a forward filtering and backward smoothing (Expectation) step, followed by a hyper-parameter estimation (Maximization) step. The model is validated across two data sets of different power exchanges. It is found that after learning a posteriori hyper-parameters, the proposed algorithm outperforms the real-time recurrent learning and the extended Kalman Filtering algorithm for recurrent networks, as well as other contemporary models that have been previously applied to the modeling of electricity spot prices.