Gaussian processes and limiting linear models

Gaussian processes retain the linear model either as a special case, or in the limit. We show how this relationship can be exploited when the data are at least partially linear. However from the perspective of the Bayesian posterior, the Gaussian processes which encode the linear model either have probability of nearly zero or are otherwise unattainable without the explicit construction of a prior with the limiting linear model in mind. We develop such a prior, and show that its practical benefits extend well beyond the computational and conceptual simplicity of the linear model. For example, linearity can be extracted on a per-dimension basis, or can be combined with treed partition models to yield a highly efficient nonstationary model. Our approach is demonstrated on synthetic and real datasets of varying linearity and dimensionality.     

Joint segmentation of multivariate Gaussian processes using mixed linear models

The joint segmentation of multiple series is considered. A mixed linear model is used to account for both covariates and correlations between signals. An estimation algorithm based on EM which involves a new dynamic programming strategy for the segmentation step is proposed. The computational efficiency of this procedure is shown and its performance is assessed through simulation experiments. Applications are presented in the field of climatic data analysis.     

Factoring Gaussian precision matrices for linear dynamic models

The linear dynamic model (LDM), also known as the Kalman filter model, has been the subject of research in the engineering, control, and more recently, machine learning and speech technology communities. The Gaussian noise processes are usually assumed to have diagonal, or occasionally full, covariance matrices. A number of recent papers have considered modelling the precision rather than covariance matrix of a Gaussian distribution, and this work applies such ideas to the LDM. A Gaussian precision matrix P can be factored into the form P = U^TSU where U is a transform and S a diagonal matrix. By varying the form of U, the covariance can be specified as being diagonal or full, or used to model a given set of spatial dependencies. Furthermore, the transform and scaling components can be shared between models, allowing richer distributions with only marginally more parameters than required to specify diagonal covariances.

The method described in this paper allows the construction of models with an appropriate number of parameters for the amount of available training data. We provide illustrative experimental results on synthetic and real speech data in which models with factored precision matrices and automatically-selected numbers of parameters are as good as or better than models with diagonal covariances on small data sets and as good as models with full covariance matrices on larger data sets.     

Nonparametric identification of population models via Gaussian processes

Population models are used to describe the dynamics of different subjects belonging to a population and play an important role in drug pharmacokinetics. A nonparametric identification scheme is proposed in which both the average impulse response of the population and the individual ones are modelled as Gaussian stochastic processes. Assuming that the average curve is an integrated Wiener process, it is shown that its estimate is a cubic spline. An empirical Bayes algorithm for estimating both the average and the individual curves is worked out. The model is tested on simulated data sets as well as on xenobiotics pharmacokinetic data.     

Fast clustering of GARCH processes via Gaussian mixture models

The financial econometrics literature includes several Multivariate GARCH models where the model parameter matrices depend on a clustering of financial assets. Those classes might be defined a priori or data-driven. When the latter approach is followed, one method for deriving asset groups is given by the use of clustering methods. In this paper, we analyze in detail one of those clustering approaches, the Gaussian mixture GARCH. This method is designed to identify groups based on the conditional variance dynamic parameters. The clustering algorithm, based on a Gaussian mixture model, has been recently proposed and is here generalized with the introduction of a correction for the presence of correlation across assets. Finally, we introduce a benchmark estimator used to assess the performances of simpler and faster estimators. Simulation experiments show evidence of the improvements given by the correction for asset correlation.     

New finite-dimensional filters for parameter estimation ofdiscrete-time linear Gaussian models

The authors derive a new class of finite-dimensional recursive filters for linear dynamical systems. The Kalman filter is a special case of their general filter. Apart from being of mathematical interest, these new finite-dimensional filters can be used with the expectation maximization (EM) algorithm to yield maximum likelihood estimates of the parameters of a linear dynamical system. Important advantages of their filter-based EM algorithm compared with the standard smoother-based EM algorithm include: 1) substantially reduced memory requirements, and 2) ease of parallel implementation on a multiprocessor system. The algorithm has applications in multisensor signal enhancement of speech signals and also econometric modeling     

Reproducing Gaussian densities and linear Gaussian detection

It is shown that the expected value of a Gaussian density function of a Gaussian random variable is another Gaussian density function. This is used to determine which of M partially observed Gaussian linear systems is the most likely, given the observations.     

On resolution and exponential discrimination between Gaussian stationary vector processes and dynamic models

We derive new necessary and sufficient conditions for exponentially convergent discrimination between two stationary vector Gaussian processes, and relate them to previously studied conditions for parameter identifiability and consistent discrimination.     

Gaussian processes and neuronal modeling

This work is a contribution towards the understanding of certain features of mathematical models of single neurons. Emphasis is set on neuronal firing, for which the first passage time (FPT) problem bears a fundamental relevance. We focus the attention on modeling the change of the neuron membrane potential between two consecutive spikes by Gaussian stochastic processes, both of Markov and of non-Markov types. Methods to solve the FPT problems, both of a theoretical and of a computational nature, are sketched, including the case of random initial values. Significant similarities or diversities between computational and theoretical results are pointed out, disclosing the role played by the correlation time that has been used to characterize the neuronal activity. It is highlighted that any conclusion on this matter is strongly model-dependent. In conclusion, an outline of the asymptotic behavior of FPT densities is provided, which is particularly useful to discuss neuronal firing under certain slow activity conditions.     

Positive real control of two-dimensional systems: Roesser models and linear repetitive processes

This paper considers the problem of positive real control for two-dimensional (2-D) discrete systems described by the Roesser model and also discrete linear repetitive processes, which are another distinct sub-class of 2-D linear systems of both systems theoretic and applications interest. The purpose of this paper is to design a dynamic output feedback controller such that the resulting closed-loop system is asymptotically stable and the closed-loop system transfer function from the disturbance to the controlled output is extended strictly positive real. We first establish a version of positive realness for 2-D discrete systems described by the Roesser state space model, then a sufficient condition for the existence of the desired output feedback controllers is obtained in terms of four LMIs. When these LMIs are feasible, an explicit parameterization of the desired output feedback controllers is given. We then apply a similar approach to discrete linear repetitive processes represented in their equivalent 1-D state-space form. Finally, we provide numerical examples to demonstrate the applicability of the approach.     

Data fusion with Gaussian processes

This paper addresses the problem of fusing multiple sets of heterogeneous sensor data using Gaussian processes (GPs). Experiments on large scale terrain modeling in mining automation are presented. Three techniques in increasing order of model complexity are discussed. The first is based on adding data to an existing GP model. The second approach treats data from different sources as different noisy samples of a common underlying terrain and fusion is performed using heteroscedastic GPs. The final approach, based on dependent GPs, models each data set by a separate GP and learns spatial correlations between data sets through auto and cross covariances. The paper presents a unifying view of approaches to data fusion using GPs, a statistical evaluation that compares these approaches and multiple previously untested variants of them and an insight into the effect of model complexity on data fusion. Experiments suggest that in situations where data being fused is not rich enough to require a complex GP data fusion model or when computational resources are limited, the use of simpler GP data fusion techniques, which are constrained versions of the more generic models, reduces optimization complexity and consequently can enable superior learning of hyperparameters, resulting in a performance gain.     

Skew Gaussian processes for classification

Machine Learning - Gaussian processes (GPs) are distributions over functions, which provide a Bayesian nonparametric approach to regression and classification. In spite of their success, GPs have...     

Fitting noncausal autoregressive signal plus noise models to noisy non-Gaussian linear processes

The problem of estimating the parameters of a noncausal autoregressive signal model from noisy observations is considered. The signal is assumed to be non-Gaussian. The measurement noise is allowed to be non-Gaussian. Two techniques that use both autocorrelations and third-order autocumulants of the data are presented for parameter estimation. Knowledge of the probability distribution of the driving noise is not required. Several simulation examples are presented to illustrate the two methods. The problem of model order selection is also addressed.     

Fuzzy set-valued Gaussian processes and Brownian motions

Gaussian processes and Brownian motion are concepts and tools in modelling important uncertain systems in many areas. In view of uncertainty complexity in many real-world problems, we extend these tools to the case where stochastic processes can take on fuzzy sets as values. In this paper, we discuss fuzzy set-valued Gaussian processes based on the results of [S. Li, Y. Ogura, V. Kreinovich, Limit Theorems and Applications of Set-Valued and Fuzzy Set-Valued Random Variables, Kluwer Academic Publishers, Dordrecht, 2002; S. Li, Y. Ogura, H.T. Nguyen, Gaussian processes and martingales for fuzzy valued variables with continuous parameter, Inform. Sci. 133 (2001) 7–21; S. Li, Y. Ogura, F.N. Proske, M.L. Puri, Central limit theorems for generalized set-valued random variables, J. Math. Anal. Appl. 285 (2003) 250–263; N.N. Lyashenko, On limit theorems for sums of independent compact random subsets in the Euclidean space, J. Soviet Math. 20 (1982) 2187–2196] and [M.L. Puri, D.A. Ralescu, The concept of normality for fuzzy random variables, Ann. Probab. 13 (1985) 1373–1379]. We also introduce the concept of fuzzy set-valued Brownian motion, and then prove several properties of such processes.     

Estimation and model predictive control of non-linear batch processes using linear parameter varying models

In this paper, feedback control is implemented for batch processes using linear models which describe the batch dynamics locally along its optimal trajectory. A Linear Parameter Varying (LPV) model obtained by interpolation between these multiple models is used to emulate the behaviour of the non-linear batch. The interpolation functions and state estimates are computed using a recursive Bayesian technique. The control technique is based on model predictive control (MPC) which is used for regulation and targeting the product specifications at the end of the batch.     

Approximation problems with the divergence criterion for Gaussian variables and Gaussian processes

System identification for stationary Gaussian processes includes an approximation problem. Currently, the subspace algorithm for this problem enjoys much attention. This algorithm is based on a transformation of a finite time series to canonical variable form followed by a truncation. There is no proof that this algorithm is the optimal solution to an approximation problem with a specific criterion. In this paper it is shown that the optimal solution to an approximation problem for Gaussian random variables with the divergence criterion is identical to the main step of the subspace algorithm. An approximation problem for stationary Gaussian processes with the divergence criterion is formulated.     

Adaptive continuous-time linear quadratic Gaussian control

The adaptive linear quadratic Gaussian control problem, where the linear transformation of the state A and the linear transformation of the control B are unknown, is solved assuming only that (A, B) is controllable and (A, Q₁^1/2) is observable, where Q ₁ determines the quadratic form for the state in the integrand of the cost functional. A weighted least squares algorithm is modified by using a random regularization to ensure that the family of estimated models is uniformly controllable and observable. A diminishing excitation is used with the adaptive control to ensure that the family of estimates is strongly consistent. A lagged certainty equivalence control using this family of estimates is shown to be self-optimizing for an ergodic, quadratic cost functional     

Maritime abnormality detection using Gaussian processes

Novelty, or abnormality, detection aims to identify patterns within data streams that do not conform to expected behaviour. This paper introduces novelty detection techniques using a combination of Gaussian processes, extreme value theory and divergence measurement to identify anomalous behaviour in both streaming and batch data. The approach is tested on both synthetic and real data, showing itself to be effective in our primary application of maritime vessel track analysis.