Statistical modeling of dissimilarity increments for d-dimensional data: Application in partitional clustering

This paper addresses the use of high order dissimilarity models in data mining problems. We explore dissimilarities between triplets of nearest neighbors, called dissimilarity increments (DIs). We derive a statistical model of DIs for d-dimensional data (d-DID) assuming that the objects follow a multivariate Gaussian distribution. Empirical evidence shows that the d-DID is well approximated by the particular case d=2. We propose the application of this model in clustering, with a partitional algorithm that uses a merge strategy on Gaussian components. Experimental results, in synthetic and real datasets, show that clustering algorithms using DID usually outperform well known clustering algorithms.     

Mixtures of Gaussian process models for human pose estimation

Discriminative human pose estimation is the problem of inferring the 3D articulated pose of a human directly from an image feature. This is a challenging problem due to the highly non-linear and multi-modal mapping from the image feature space to the pose space. To address this problem, we propose a model employing a mixture of Gaussian processes where each Gaussian process models a local region of the pose space. By employing the models in this way we are able to overcome the limitations of Gaussian processes applied to human pose estimation — their O(N³) time complexity and their uni-modal predictive distribution. Our model is able to give a multi-modal predictive distribution where each mode is represented by a different Gaussian process prediction. A logistic regression model is used to give a prior over each expert prediction in a similar fashion to previous mixture of expert models. We show that this technique outperforms existing state of the art regression techniques on human pose estimation data sets for ballet dancing, sign language and the HumanEva data set.     

Privacy-aware collection of aggregate spatial data

Privacy concerns can be a major barrier to collecting aggregate data from the public. Recent research proposes negative surveys that collect negative data, which is complementary to the true data. This opens a new direction for privacy-aware data collection. However, the existing approach cannot avoid certain errors when applied to many spatial data collection tasks. The errors can make the data unusable in many real scenarios. We propose Gaussian negative surveys. We modulate data collection based on Gaussian distribution. The collected data can be used to compute accurate spatial distribution of participants and can be used to accurately answer range aggregate queries. Our approach avoids the errors that can occur with the existing approach. Our experiments show that we achieve an excellent balance between privacy and accuracy.     

Distributed robust Gaussian Process regression

We study distributed and robust Gaussian Processes where robustness is introduced by a Gaussian Process prior on the function values combined with a Student-t likelihood. The posterior distribution is approximated by a Laplace Approximation, and together with concepts from Bayesian Committee Machines, we efficiently distribute the computations and render robust GPs on huge data sets feasible. We provide a detailed derivation and report on empirical results. Our findings on real and artificial data show that our approach outperforms existing baselines in the presence of outliers by using all available data.     

Focused multi-task learning in a Gaussian process framework

Multi-task learning, learning of a set of tasks together, can improve performance in the individual learning tasks. Gaussian process models have been applied to learning a set of tasks on different data sets, by constructing joint priors for functions underlying the tasks. In these previous Gaussian process models, the setting has been symmetric in the sense that all the tasks have been assumed to be equally important, whereas in settings such as transfer learning the goal is asymmetric, to enhance performance in a target task given the other tasks. We propose a focused Gaussian process model which introduces an ??explaining away?? model for each of the additional tasks to model their non-related variation, in order to focus the transfer to the task-of-interest. This focusing helps reduce the key problem of negative transfer, which may cause performance to even decrease if the tasks are not related closely enough. In experiments, our model improves performance compared to single-task learning, symmetric multi-task learning using hierarchical Dirichlet processes, transfer learning based on predictive structure learning, and symmetric multi-task learning with Gaussian processes.     

Clustering data with measurement errors

Traditional clustering methods assume that there is no measurement error, or uncertainty, associated with data. Often, however, real world applications require treatment of data that have such errors. In the presence of measurement errors, well-known clustering methods like k-means and hierarchical clustering may not produce satisfactory results.In this article, we develop a statistical model and algorithms for clustering data in the presence of errors. We assume that the errors associated with data follow a multivariate Gaussian distribution and are independent between data points. The model uses the maximum likelihood principle and provides us with a new metric for clustering. This metric is used to develop two algorithms for error-based clustering, hError and kError, that are generalizations of Ward's hierarchical and k-means clustering algorithms, respectively.We discuss types of clustering problems where error information associated with the data to be clustered is readily available and where error-based clustering is likely to be superior to clustering methods that ignore error. We focus on clustering derived data (typically parameter estimates) obtained by fitting statistical models to the observed data. We show that, for Gaussian distributed observed data, the optimal error-based clusters of derived data are the same as the maximum likelihood clusters of the observed data. We also report briefly on two applications with real-world data and a series of simulation studies using four statistical models: (1) sample averaging, (2) multiple linear regression, (3) ARIMA models for time-series, and (4) Markov chains, where error-based clustering performed significantly better than traditional clustering methods.     

Scale-invariant clustering with minimum volume ellipsoids

This paper develops theory and algorithms concerning a new metric for clustering data. The metric minimizes the total volume of clusters, where the volume of a cluster is defined as the volume of the minimum volume ellipsoid (MVE) enclosing all data points in the cluster. This metric is scale-invariant, that is, the optimal clusters are invariant under an affine transformation of the data space. We introduce the concept of outliers in the new metric and show that the proposed method of treating outliers asymptotically recovers the data distribution when the data comes from a single multivariate Gaussian distribution. Two heuristic algorithms are presented that attempt to optimize the new metric. On a series of empirical studies with Gaussian distributed simulated data, we show that volume-based clustering outperforms well-known clustering methods such as k-means, Ward's method, SOM, and model-based clustering.     

Clustering data with measurement errors

Traditional clustering methods assume that there is no measurement error, or uncertainty, associated with data. Often, however, real world applications require treatment of data that have such errors. In the presence of measurement errors, well-known clustering methods like k-means and hierarchical clustering may not produce satisfactory results.In this article, we develop a statistical model and algorithms for clustering data in the presence of errors. We assume that the errors associated with data follow a multivariate Gaussian distribution and are independent between data points. The model uses the maximum likelihood principle and provides us with a new metric for clustering. This metric is used to develop two algorithms for error-based clustering, hError and kError, that are generalizations of Ward's hierarchical and k-means clustering algorithms, respectively.We discuss types of clustering problems where error information associated with the data to be clustered is readily available and where error-based clustering is likely to be superior to clustering methods that ignore error. We focus on clustering derived data (typically parameter estimates) obtained by fitting statistical models to the observed data. We show that, for Gaussian distributed observed data, the optimal error-based clusters of derived data are the same as the maximum likelihood clusters of the observed data. We also report briefly on two applications with real-world data and a series of simulation studies using four statistical models: (1) sample averaging, (2) multiple linear regression, (3) ARIMA models for time-series, and (4) Markov chains, where error-based clustering performed significantly better than traditional clustering methods.     

Uncertainty-based learning of acoustic models from noisy data

We consider the problem of acoustic modeling of noisy speech data, where the uncertainty over the data is given by a Gaussian distribution. While this uncertainty has been exploited at the decoding stage via uncertainty decoding, its usage at the training stage remains limited to static model adaptation. We introduce a new expectation maximization (EM) based technique, which we call uncertainty training, that allows us to train Gaussian mixture models (GMMs) or hidden Markov models (HMMs) directly from noisy data with dynamic uncertainty. We evaluate the potential of this technique for a GMM-based speaker recognition task on speech data corrupted by real-world domestic background noise, using a state-of-the-art signal enhancement technique and various uncertainty estimation techniques as a front-end. Compared to conventional training, the proposed training algorithm results in 3–4% absolute improvement in speaker recognition accuracy by training from either matched, unmatched or multi-condition noisy data. This algorithm is also applicable with minor modifications to maximum a posteriori (MAP) or maximum likelihood linear regression (MLLR) acoustic model adaptation from noisy data and to other data than audio.     

Features modeling with an α-stable distribution: Application to pattern recognition based on continuous belief functions

The aim of this paper is to show the interest in fitting features with an α-stable distribution to classify imperfect data. The supervised pattern recognition is thus based on the theory of continuous belief functions, which is a way to consider imprecision and uncertainty of data. The distributions of features are supposed to be unimodal and estimated by a single Gaussian and α-stable model. Experimental results are first obtained from synthetic data by combining two features of one dimension and by considering a vector of two features. Mass functions are calculated from plausibility functions by using the generalized Bayes theorem. The same study is applied to the automatic classification of three types of sea floor (rock, silt and sand) with features acquired by a mono-beam echo-sounder. We evaluate the quality of the α-stable model and the Gaussian model by analyzing qualitative results, using a Kolmogorov–Smirnov test (K–S test), and quantitative results with classification rates. The performances of the belief classifier are compared with a Bayesian approach.     

Continuous mixture modeling via goodness-of-fit ridges

We present a novel method for representing “extruded” distributions. An extruded distribution is an M-dimensional manifold in the parameter space of the component distribution. Representations of that manifold are “continuous mixture models”. We present a method for forming one-dimensional continuous Gaussian mixture models of sampled extruded Gaussian distributions via ridges of goodness-of-fit. Using Monte Carlo simulations and ROC analysis, we explore the utility of a variety of binning techniques and goodness-of-fit functions. We demonstrate that extruded Gaussian distributions are more accurately and consistently represented by continuous Gaussian mixture models than by finite Gaussian mixture models formed via maximum likelihood expectation maximization.     

Adaptive filtering of a random signal in Gaussian white noise

We consider the problem of estimating an infinite-dimensional vector θ observed in Gaussian white noise. Under the condition that components of the vector have a Gaussian prior distribution that depends on an unknown parameter β, we construct an adaptive estimator with respect to β. The proposed method of estimation is based on the empirical Bayes approach.     

Deep recurrent Gaussian processes for outlier-robust system identification

Gaussian Processes (GP) comprise a powerful kernel-based machine learning paradigm which has recently attracted the attention of the nonlinear system identification community, specially due to its inherent Bayesian-style treatment of the uncertainty. However, since standard GP models assume a Gaussian distribution for the observation noise, i.e., a Gaussian likelihood, the learning and predictive capabilities of such models can be severely degraded when outliers are present in the data. In this paper, motivated by our previous work on GP learning with data containing outliers and recent advances in hierarchical (deep GPs) and recurrent GP (RGP) approaches, we introduce an outlier-robust recurrent GP model, the RGP-t. Our approach explicitly models the observation layer, which includes a heavy-tailed Student-t likelihood, and allows for a hierarchy of multiple transition layers to learn the system dynamics directly from estimation data contaminated by outliers. In addition, we modify the original variational framework of standard RGP in order to perform inference with the new RGP-t model. The proposed approach is comprehensively evaluated using six artificial benchmarks, within several outlier contamination levels, and two datasets related to process industry systems (pH neutralization and heat exchanger), whose estimation data undergo large contamination rates. The simulation results obtained by the RGP-t model indicates an impressive resilience to outliers and a superior capability to learn nonlinear dynamics directly from highly outlier-contaminated data in comparison to existing GP models.     

Lattice-Based High-Dimensional Gaussian Filtering and the Permutohedral Lattice

High-dimensional Gaussian filtering is a popular technique in image processing, geometry processing and computer graphics for smoothing data while preserving important features. For instance, the bilateral filter, cross bilateral filter and non-local means filter fall under the broad umbrella of high-dimensional Gaussian filters. Recent algorithmic advances therein have demonstrated that by relying on a sampled representation of the underlying space, one can obtain speed-ups of orders of magnitude over the naïve approach. The simplest such sampled representation is a lattice, and it has been used successfully in the bilateral grid and the permutohedral lattice algorithms. In this paper, we analyze these lattice-based algorithms, developing a general theory of lattice-based high-dimensional Gaussian filtering. We consider the set of criteria for an optimal lattice for filtering, as it offers a good tradeoff of quality for computational efficiency, and evaluate the existing lattices under the criteria. In particular, we give a rigorous exposition of the properties of the permutohedral lattice and argue that it is the optimal lattice for Gaussian filtering. Lastly, we explore further uses of the permutohedral-lattice-based Gaussian filtering framework, showing that it can be easily adapted to perform mean shift filtering and yield improvement over the traditional approach based on a Cartesian grid.     

Image reconstruction from scattered Radon data by weighted positive definite kernel functions

We propose a novel kernel-based method for image reconstruction from scattered Radon data. To this end, we employ generalized Hermite–Birkhoff interpolation by positive definite kernel functions. For radial kernels, however, a straightforward application of the generalized Hermite–Birkhoff interpolation method fails to work, as we prove in this paper. To obtain a well-posed reconstruction scheme for scattered Radon data, we introduce a new class of weighted positive definite kernels, which are symmetric but not radially symmetric. By our construction, the resulting weighted kernels are combinations of radial positive definite kernels and positive weight functions. This yields very flexible image reconstruction methods, which work for arbitrary distributions of Radon lines. We develop suitable representations for the weighted basis functions and the symmetric positive definite kernel matrices that are resulting from the proposed reconstruction scheme. For the relevant special case, where Gaussian radial kernels are combined with Gaussian weights, explicit formulae for the weighted Gaussian basis functions and the kernel matrices are given. Supporting numerical examples are finally presented.     

Gaussian classical-quantum channels: Gain from entanglement-assistance

In the present paper we introduce and study Bosonic Gaussian classical-quantum (c-q) channels; embedding of the classical input in quantum input is always possible, and therefore the classical entanglement-assisted capacity C _ea under an appropriate input constraint is well defined. We prove the general property of entropy increase for a weak complementary channel, which implies the equality C = C _ea (where C is the unassisted capacity) for a certain class of c-q Gaussian channels under an appropriate energy-type constraint. On the other hand, we show by an explicit example that the inequality C < C _ea is not unusual for constrained c-q Gaussian channel.     

Distributed data clustering in sensor networks

Low overhead analysis of large distributed data sets is necessary for current data centers and for future sensor networks. In such systems, each node holds some data value, e.g., a local sensor read, and a concise picture of the global system state needs to be obtained. In resource-constrained environments like sensor networks, this needs to be done without collecting all the data at any location, i.e., in a distributed manner. To this end, we address the distributed clustering problem, in which numerous interconnected nodes compute a clustering of their data, i.e., partition these values into multiple clusters, and describe each cluster concisely. We present a generic algorithm that solves the distributed clustering problem and may be implemented in various topologies, using different clustering types. For example, the generic algorithm can be instantiated to cluster values according to distance, targeting the same problem as the famous k-means clustering algorithm. However, the distance criterion is often not sufficient to provide good clustering results. We present an instantiation of the generic algorithm that describes the values as a Gaussian Mixture (a set of weighted normal distributions), and uses machine learning tools for clustering decisions. Simulations show the robustness, speed and scalability of this algorithm. We prove that any implementation of the generic algorithm converges over any connected topology, clustering criterion and cluster representation, in fully asynchronous settings.     

A novel split-and-merge algorithm for hierarchical clustering of Gaussian mixture models

The paper presents a novel split-and-merge algorithm for hierarchical clustering of Gaussian mixture models, which tends to improve on the local optimal solution determined by the initial constellation. It is initialized by local optimal parameters obtained by using a baseline approach similar to k-means, and it tends to approach more closely to the global optimum of the target clustering function, by iteratively splitting and merging the clusters of Gaussian components obtained as the output of the baseline algorithm. The algorithm is further improved by introducing model selection in order to obtain the best possible trade-off between recognition accuracy and computational load in a Gaussian selection task applied within an actual recognition system. The proposed method is tested both on artificial data and in the framework of Gaussian selection performed within a real continuous speech recognition system, and in both cases an improvement over the baseline method has been observed.     

Empirical and kernel estimation of covariate distribution conditional on survival time

In biomedical research there is often interest in describing covariate distributions given different survival groups. This is not immediately available due to censoring. In this paper we develop an empirical estimate of the conditional covariate distribution under the proportional hazards regression model. We show that it converges weakly to a Gaussian process and provide its variance estimate. We then apply kernel smoothing to obtain an estimate of the corresponding density function. The density estimate is consistent and has the same rate of convergence as the classical kernel density estimator. We have developed an R package to implement our methodology, which is demonstrated through the Mayo Clinic primary biliary cirrhosis data.     

On Detection of Gaussian Stochastic Sequences

We consider the minimax detection problem for a Gaussian random signal vector in white Gaussian additive noise. It is assumed that an unknown vector σ of signal vector intensities belongs to a given set ε. We investigate when it is possible to replace the set ε with a smaller set ε₀ without loss of quality (and, in particular, replace it with a single point σ₀).