A reservoir-driven non-stationary hidden Markov model

In this work, we propose a novel approach towards sequential data modeling that leverages the strengths of hidden Markov models and echo-state networks (ESNs) in the context of non-parametric Bayesian inference approaches. We introduce a non-stationary hidden Markov model, the time-dependent state transition probabilities of which are driven by a high-dimensional signal that encodes the whole history of the modeled observations, namely the state vector of a postulated observations-driven ESN reservoir. We derive an efficient inference algorithm for our model under the variational Bayesian paradigm, and we examine the efficacy of our approach considering a number of sequential data modeling applications.     

Regularized variational Bayesian learning of echo state networks with delay&sum readout

In this work, a variational Bayesian framework for efficient training of echo state networks (ESNs) with automatic regularization and delay&sum (D&S) readout adaptation is proposed. The algorithm uses a classical batch learning of ESNs. By treating the network echo states as fixed basis functions parameterized with delay parameters, we propose a variational Bayesian ESN training scheme. The variational approach allows for a seamless combination of sparse Bayesian learning ideas and a variational Bayesian space-alternating generalized expectation-maximization (VB-SAGE) algorithm for estimating parameters of superimposed signals. While the former method realizes automatic regularization of ESNs, which also determines which echo states and input signals are relevant for "explaining" the desired signal, the latter method provides a basis for joint estimation of D&S readout parameters. The proposed training algorithm can naturally be extended to ESNs with fixed filter neurons. It also generalizes the recently proposed expectation-maximization-based D&S readout adaptation method. The proposed algorithm was tested on synthetic data prediction tasks as well as on dynamic handwritten character recognition.     

A normal copula model for the arrival process in a call center

We propose and examine a probabilistic model for the multivariate distribution of the number of calls in each period of the day (e.g., 15 or 30 min) in a call center, where the marginal distribution of the number of calls in any given period is arbitrary, and the dependence between the periods is modeled via a normal copula. Conditional on the number of calls in a period, their arrival times are independent and uniformly distributed over the period. This type of model has the advantage of being simple and reasonably flexible, and can match the correlations between the arrival counts in different periods much better than previously proposed models. For the situation where the number of periods is large, so the number of correlations to estimate can be excessive, we propose simple parametric forms for the correlations, defined as functions of the time lag between the periods. We test our proposed models on three data sets taken from real‐life call centers and compare their goodness of fit to the best previously proposed methods that we know. In the three cases, the new models provide a much better match of the correlations and coefficients of variation of the arrival counts in individual periods.     

Evaluating perceived voice quality on packet networks using different random neural network architectures

Voice over Internet Protocol (VoIP) is one of the fastest growing technologies in the world. In VoIP speech signals are transmitted over the same network used for data communications. The internet is not a robust network and is subjected to delay, jitter, and packet loss. It is very important to measure and monitor the quality of service (QoS) the users experience in VoIP networks; this is not an easy task and usually requires subjective tests. In this paper we have analyzed three non-intrusive models to measure and monitor voice quality using Random Neural Networks (RNN). A RNN is an open queuing network with positive and negative signals. We have assessed the voice quality based on various parameters i.e. delay, jitter, packet loss, and codec. In our approach we have used the Mean Opinion Score (MOS) calculated using a Perceptual Evaluation of Speech Quality (PESQ) algorithm to generate data for training the RNN model. We have studied two feed-forward models and a recurrent architecture. We have found that the simple feed-forward architecture has produced the most accurate results compared to the other two architectures.     

Financial volatility trading using recurrent neural networks

We simulate daily trading of straddles on financial indexes. The straddles are traded based on predictions of daily volatility differences in the indexes. The main predictive models studied are recurrent neural nets (RNN). Such applications have often been studied in isolation. However, due to the special character of daily financial time-series, it is difficult to make full use of RNN representational power. Recurrent networks either tend to overestimate noisy data, or behave like finite-memory sources with shallow memory; they hardly beat classical fixed-order Markov models. To overcome data nonstationarity, we use a special technique that combines sophisticated models fitted on a larger data set, with a fixed set of simple-minded symbolic predictors using only recent inputs. Finally, we compare our predictors with the GARCH family of econometric models designed to capture time-dependent volatility structure in financial returns. GARCH models have been used to trade volatility. Experimental results show that while GARCH models cannot generate any significantly positive profit, by careful use of recurrent networks or Markov models, the market makers can generate a statistically significant excess profit, but then there is no reason to prefer RNN over much more simple and straightforward Markov models. We argue that any report containing RNN results on financial tasks should be accompanied by results achieved by simple finite-memory sources combined with simple techniques to fight nonstationarity in the data.     

A new multi-level statistical model for variable bit rate MPEG sources over ATM networks and its performance study

In this paper, we propose a new traffic model for MPEG-coded video sequences. The proposed modeling scheme uses scene-based traffic characteristics and considers the correlations between frames of consecutive group of pictures (GOPs). Using a simple scene detection algorithm, scene changes are modeled by a state transition matrix and the number of GOPs of a scene state is modeled by a geometric distribution. Frames of a scene are modeled by the mean I, P, and B frame sizes of each state. For more accurate traffic modeling, the residual bits that represent the difference between the original frame size and the mean frame size of each frame type are compensated by autoregressive processes. The modeling results show that our scene-based model can capture the statistical traffic characteristics of the original video sequences well and estimate the queueing performance with good approximation quality.     

Heuristic Dynamic Programming Using Echo State Network For Multivariable Tracking Control Of Wastewater Treatment Process

This paper proposes a heuristic dynamic programming (HDP) scheme to simultaneously control the dissolved oxygen concentration and the nitrate level in wastewater treatment processes (WWTP). Unlike traditional HDP schemes, the optimal control values are calculated in an analytical way by the proposed HDP controller. It can reduce the learning burden of the HDP controller to a great extent. The system model and the evaluation index J are approximated by two echo state networks (ESNs). Gradient‐based learning algorithms are employed to train both ESNs online, and the convergence of the training algorithm is investigated based on Lyapunov theory. The performance of the proposed ESN‐based HDP (E‐HDP) controller is tested and evaluated on a WWTP benchmark. Experimental results demonstrate that the proposed approach can achieve effective performance.     

基于自校正回归神经元网络的预报建模

讨论了回归神经元网络（ＲＮＮ）的网络结构和基本实现方法，提出了主元分析（ＰＣＡ）和具有自校正功能的回归神经元网络相结合的非线性时变系统预报建模方法，并用于减压塔塔顶温度的预报．结果表明，该方法具有良好的预报性能．     

Training echo state networks for rotation-invariant bone marrow cell classification

The main principle of diagnostic pathology is the reliable interpretation of individual cells in context of the tissue architecture. Especially a confident examination of bone marrow specimen is dependent on a valid classification of myeloid cells. In this work, we propose a novel rotation-invariant learning scheme for multi-class echo state networks (ESNs), which achieves very high performance in automated bone marrow cell classification. Based on representing static images as temporal sequence of rotations, we show how ESNs robustly recognize cells of arbitrary rotations by taking advantage of their short-term memory capacity. The performance of our approach is compared to a classification random forest that learns rotation-invariance in a conventional way by exhaustively training on multiple rotations of individual samples. The methods were evaluated on a human bone marrow image database consisting of granulopoietic and erythropoietic cells in different maturation stages. Our ESN approach to cell classification does not rely on segmentation of cells or manual feature extraction and can therefore directly be applied to image data.

     

Recurrent neural network technique for behavioral modeling of power amplifier with memory effects

A new technique for behavioral modeling of power amplifier (PA) with short‐ and long‐term memory effects is presented here using recurrent neural networks (RNNs). RNN can be trained directly with only the input–output data without having to know the internal details of the circuit. The trained models can reflect the behavior of nonlinear circuits. In our proposed technique, we extract slow‐changing signals from the inputs and outputs of the PA and use these signals as extra inputs of RNN model to effectively represent long‐term memory effects. The methodology using the proposed RNN for modeling short‐term and long‐term memory effects is discussed. Examples of behavioral modeling of PAs with short‐ and long‐term memory using both the existing dynamic neural networks and the proposed RNNs techniques are shown. © 2014 Wiley Periodicals, Inc. Int J RF and Microwave CAE 25:289–298, 2015.     

Reverse Nearest Neighbors Search in Ad Hoc Subspaces

Given an object q, modeled by a multidimensional point, a reverse nearest neighbors (RNN) query returns the set of objects in the database that have q as their nearest neighbor. In this paper, we study an interesting generalization of the RNN query, where not all dimensions are considered, but only an ad hoc subset thereof. The rationale is that 1) the dimensionality might be too high for the result of a regular RNN query to be useful, 2) missing values may implicitly define a meaningful subspace for RNN retrieval, and 3) analysts may be interested in the query results only for a set of (ad hoc) problem dimensions (i.e., object attributes). We consider a suitable storage scheme and develop appropriate algorithms for projected RNN queries, without relying on multidimensional indexes. Given the significant cost difference between random and sequential data accesses, our algorithms are based on applying sequential accesses only on the projected atomic values of the data at each dimension, to progressively derive a set of RNN candidates. Whether these candidates are actual RNN results is then validated via an optimized refinement step. In addition, we study variants of the projected RNN problem, including RkNN search, bichromatic RNN, and RNN retrieval for the case where sequential accesses are not possible. Our methods are experimentally evaluated with real and synthetic data     

Copula density estimation by total variation penalized likelihood with linear equality constraints

A copula density is the joint probability density function (PDF) of a random vector with uniform marginals. An approach to bivariate copula density estimation is introduced that is based on maximum penalized likelihood estimation (MPLE) with a total variation (TV) penalty term. The marginal unity and symmetry constraints for copula density are enforced by linear equality constraints. The TV-MPLE subject to linear equality constraints is solved by an augmented Lagrangian and operator-splitting algorithm. It offers an order of magnitude improvement in computational efficiency over another TV-MPLE method without constraints solved by the log-barrier method for the second order cone program. A data-driven selection of the regularization parameter is through K-fold cross-validation (CV). Simulation and real data application show the effectiveness of the proposed approach. The MATLAB code implementing the methodology is available online.     

Recurrent kernel machines: computing with infinite echo state networks

Echo state networks (ESNs) are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network computes a broad set of nonlinear, spatiotemporal mappings of the input data, on which linear regression or classification can easily be performed. One could consider the reservoir as a spatiotemporal kernel, in which the mapping to a high-dimensional space is computed explicitly. In this letter, we build on this idea and extend the concept of ESNs to infinite-sized recurrent neural networks, which can be considered recursive kernels that subsequently can be used to create recursive support vector machines. We present the theoretical framework, provide several practical examples of recursive kernels, and apply them to typical temporal tasks.     

随机神经网络发展现状综述

随机神经网络 (RNN)在人工神经网络中是一类比较独特、出现较晚的神经网络 ,它的网络结构、学习算法、状态更新规则以及应用等方面都因此具有自身的特点 .作为仿生神经元数学模型 ,随机神经网络在联想记忆、图像处理、组合优化问题上都显示出较强的优势 .在阐述随机神经网络发展现状、网络特性以及广泛应用的同时 ,专门将RNN分别与Hopfield网络、模拟退火算法和Boltzmann机在组合优化问题上的应用进行了分析对比 ,指出RNN是解决旅行商 (TSP)等问题的有效途径     

Comparison of semiparametric maximum likelihood estimation and two-stage semiparametric estimation in copula models

We consider bivariate distributions that are specified in terms of a parametric copula function and nonparametric or semiparametric marginal distributions. The performance of two semiparametric estimation procedures based on censored data is discussed: maximum likelihood (ML) and two-stage pseudolikelihood (PML) estimation. The two-stage procedure involves less computation and it is of interest to see whether it is significantly less efficient than the full maximum likelihood approach. We also consider cases where the copula model is misspecified, in which case PML may be better. Extensive simulation studies demonstrate that in the absence of covariates, two-stage estimation is highly efficient and has significant robustness advantages for estimating marginal distributions. In some settings, involving covariates and a high degree of association between responses, ML is more efficient. For the estimation of association, PML does not offer an advantage.     

Training recurrent neural networks using a hybrid algorithm

This paper proposes a new hybrid approach for recurrent neural networks (RNN). The basic idea of this approach is to train an input layer by unsupervised learning and an output layer by supervised learning. In this method, the Kohonen algorithm is used for unsupervised learning, and dynamic gradient descent method is used for supervised learning. The performances of the proposed algorithm are compared with backpropagation through time (BPTT) on three benchmark problems. Simulation results show that the performances of the new proposed algorithm exceed the standard backpropagation through time in the reduction of the total number of iterations and in the learning time required in the training process.     

Design and implementation of a random neural network routing engine

Random neural network (RNN) is an analytically tractable spiked neural network model that has been implemented in software for a wide range of applications for over a decade. This paper presents the hardware implementation of the RNN model. Recently, cognitive packet networks (CPN) is proposed as an alternative packet network architecture where there is no routing table, instead the RNN based reinforcement learning is used to route packets. Particularly, we describe implementation details for the RNN based routing engine of a CPN network processor chip: the smart packet processor (SPP). The SPP is a dual port device that stores, modifies, and interprets the defining characteristics of multiple RNN models. In addition to hardware design improvements over the software implementation such as the dual access memory, output calculation step, and reduced output calculation module, this paper introduces a major modification to the reinforcement learning algorithm used in the original CPN specification such that the number of weight terms are reduced from 2n/sup 2/ to 2n. This not only yields significant memory savings, but it also simplifies the calculations for the steady state probabilities (neuron outputs in RNN). Simulations have been conducted to confirm the proper functionality for the isolated SPP design as well as for the multiple SPP's in a networked environment.     

SimQPN—A tool and methodology for analyzing queueing Petri net models by means of simulation

The queueing Petri net (QPN) paradigm provides a number of benefits over conventional modeling paradigms such as queueing networks and generalized stochastic Petri nets. Using queueing Petri nets (QPNs), one can integrate both hardware and software aspects of system behavior into the same model. This lends itself very well to modeling distributed component-based systems, such as modern e-business applications. However, currently available tools and techniques for QPN analysis suffer the state space explosion problem, imposing a limit on the size of the models that are tractable. In this paper, we present SimQPN—a simulation tool for QPNs that provides an alternative approach to analyze QPN models, circumventing the state space explosion problem. In doing this, we propose a methodology for analyzing QPN models by means of discrete event simulation. The methodology shows how to simulate QPN models and analyze the output data from simulation runs. We validate our approach by applying it to study several different QPN models, ranging from simple models to models of realistic systems. The performance of point and interval estimators implemented in SimQPN is subjected to a rigorous experimental analysis.     

Using artificial neural networks to enhance CART

Accuracy is a critical factor in predictive modeling. A predictive model such as a decision tree must be accurate to draw conclusions about the system being modeled. This research aims at analyzing and improving the performance of classification and regression trees (CART), a decision tree algorithm, by evaluating and deriving a new methodology based on the performance of real-world data sets that were studied. This paper introduces a new approach to tree induction to improve the efficiency of the CART algorithm by combining the existing functionality of CART with the addition of artificial neural networks (ANNs). Trained ANNs are utilized by the tree induction algorithm by generating new, synthetic data, which have been shown to improve the overall accuracy of the decision tree model when actual training samples are limited. In this paper, traditional decision trees developed by the standard CART methodology are compared with the enhanced decision trees that utilize the ANN’s synthetic data generation, or CART+. This research demonstrates the improved accuracies that can be obtained with CART+, which can ultimately improve the knowledge that can be extracted by researchers about a system being modeled.     

Multi-agent modeling of multiple FX-markets by neural networks

We introduce an explanatory multi-agent approach of multiple FX-market modeling based on neural networks. We consider the explicit and implicit dynamics of the market price. This paper extends previous work of modeling a single FX-market to an integrated approach, which allows one to treat several FX-markets simultaneously. Our approach is based on feedforward neural networks. Neural networks allow the fitting of high-dimensional nonlinear models, which is often utilized in econometrics. Merging the economic theory of multi-agents with neural networks, our model concerns semantic specifications instead of being limited to ad hoc functional relationships. As an advantage, our multi-agent model allows one to fit the behavior of real-world financial data. We exemplify the USD/DEM and YEN/DEM FX-Market simultaneously. Fitting real-world data, our approach is superior to more conventional forecasting techniques.