Parameter estimation of two-level nonlinear mixed effects models using first order conditional linearization and the EM algorithm

Multi-level nonlinear mixed effects (ML-NLME) models have received a great deal of attention in recent years because of the flexibility they offer in handling the repeated-measures data arising from various disciplines. In this study, we propose both maximum likelihood and restricted maximum likelihood estimations of ML-NLME models with two-level random effects, using first order conditional expansion (FOCE) and the expectation–maximization (EM) algorithm. The FOCE–EM algorithm was compared with the most popular Lindstrom and Bates (LB) method in terms of computational and statistical properties. Basal area growth series data measured from Chinese fir (Cunninghamia lanceolata) experimental stands and simulated data were used for evaluation. The FOCE–EM and LB algorithms given the same parameter estimates and fit statistics for models that converged by both. However, FOCE–EM converged for all the models, while LB did not, especially for the models in which two-level random effects are simultaneously considered in several base parameters to account for between-group variation. We recommend the use of FOCE–EM in ML-NLME models, particularly when convergence is a concern in model selection.     

Recognizing multi-modal sensor signals using evolutionary learning of dynamic Bayesian networks

Multi-modal context-aware systems can provide user-adaptive services, but it requires complicated recognition models with larger resources. The limitations to build optimal models and infer the context efficiently make it difficult to develop practical context-aware systems. We developed a multi-modal context-aware system with various wearable sensors including accelerometers, gyroscopes, physiological sensors, and data gloves. The system used probabilistic models to handle the uncertain and noisy time-series sensor data. In order to construct the efficient probabilistic models, this paper uses an evolutionary algorithm to model structure and EM algorithm to determine parameters. The trained models are selectively inferred based on a semantic network which describes the semantic relations of the contexts and sensors. Experiments with the real data collected show the usefulness of the proposed method.     

Online learning with hidden markov models

We present an online version of the expectation-maximization (EM) algorithm for hidden Markov models (HMMs). The sufficient statistics required for parameters estimation is computed recursively with time, that is, in an online way instead of using the batch forward-backward procedure. This computational scheme is generalized to the case where the model parameters can change with time by introducing a discount factor into the recurrence relations. The resulting algorithm is equivalent to the batch EM algorithm, for appropriate discount factor and scheduling of parameters update. On the other hand, the online algorithm is able to deal with dynamic environments, i.e., when the statistics of the observed data is changing with time. The implications of the online algorithm for probabilistic modeling in neuroscience are briefly discussed.     

一种基于高阶累积量的因果结构学习算法

从观测数据中学习因果结构具有重要的应用价值。目前,一类学习因果结构的方法是基于函数因果模型假设,通过检验噪声与原因变量的独立性来学习因果结构。然而,该类方法涉及高计算复杂度的独立性检验过程,影响结构学习算法的实用性和鲁棒性。为此,提出了一种在线性非高斯模型下,利用高阶累积量作为独立性评估的因果结构学习算法。该算法主要分为两个步骤,第一个步骤是利用基于条件独立性约束的方法学习到因果结构的马尔可夫等价类,第二个步骤是定义了一种基于高阶累积量的得分,该得分可以判别两个随机变量的独立性,从而可以从马尔可夫等价类中搜索到最佳独立性得分的因果结构作为算法的输出。该算法的优势在于：a)相比基于核方法的独立性检验,该方法有较低的计算复杂度;b)基于得分搜索的方法,可以得到一个最匹配数据生成过程的模型,提高学习方法的鲁棒性。实验结果表明,基于高阶累积量的因果结构学习方法在合成数据中F₁得分提高了5%,并在真实数据中学习到更多的因果方向。     

NAT model–based compression of Bayesian network CPTs over multivalued variables

Nonimpeding noisy‐AND tree (NAT) models offer a highly expressive approximate representation for significantly reducing the space of Bayesian networks (BNs). They also improve efficiency of BN inference significantly. To enable these advantages for general BNs, several technical advancements are made in this work to compress target BN conditional probability tables (CPTs) over multivalued variables into NAT models. We extend the semantics of NAT models beyond graded variables that causal independence models commonly adhered to and allow NAT modeling in nominal causal variables. We overcome the limitation of well‐defined pairwise causal interaction (PCI) bits and present a flexible PCI pattern extraction from target CPTs. We extend parameter estimation for binary NAT models to constrained gradient descent for compressing target CPTs over multivalued variables. We reveal challenges associated with persistent leaky causes and develop a novel framework for PCI pattern extraction when persistent leaky causes exist. The effectiveness of the CPT compression is validated experimentally.     

Parameter Estimation of a State-Space Model of Noise for Robust Speech Recognition

In this paper, parameter estimation of a state-space model of noise or noisy speech cepstra is investigated. A blockwise EM algorithm is derived for the estimation of the state and observation noise covariance from noise-only input data. It is supposed to be used during the offline training mode of a speech recognizer. Further a sequential online EM algorithm is developed to adapt the observation noise covariance on noisy speech cepstra at its input. The estimated parameters are then used in model-based speech feature enhancement for noise-robust automatic speech recognition. Experiments on the AURORA4 database lead to improved recognition results with a linear state model compared to the assumption of stationary noise.      

A graphical model for audiovisual object tracking

We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment.     

Uncertainty-based learning of acoustic models from noisy data

We consider the problem of acoustic modeling of noisy speech data, where the uncertainty over the data is given by a Gaussian distribution. While this uncertainty has been exploited at the decoding stage via uncertainty decoding, its usage at the training stage remains limited to static model adaptation. We introduce a new expectation maximization (EM) based technique, which we call uncertainty training, that allows us to train Gaussian mixture models (GMMs) or hidden Markov models (HMMs) directly from noisy data with dynamic uncertainty. We evaluate the potential of this technique for a GMM-based speaker recognition task on speech data corrupted by real-world domestic background noise, using a state-of-the-art signal enhancement technique and various uncertainty estimation techniques as a front-end. Compared to conventional training, the proposed training algorithm results in 3–4% absolute improvement in speaker recognition accuracy by training from either matched, unmatched or multi-condition noisy data. This algorithm is also applicable with minor modifications to maximum a posteriori (MAP) or maximum likelihood linear regression (MLLR) acoustic model adaptation from noisy data and to other data than audio.     

基于SMC-PHDF的部分可分辨的群目标跟踪算法

提出一种基于粒子概率假设密度滤波器(Sequential Monte Carlo probability hypothesis density filter, SMC-PHDF)的部分可分辨的群目标跟踪算法. 该算法可直接获得群而非个体的个数和状态估计. 这里群的状态包括群的质心状态和形状. 为了估计群的个数和状态, 该算法利用高斯混合模型(Gaussian mixture models, GMM)拟合SMC-PHDF中经重采样后的粒子分布, 这里混合模型的元素个数和参数分别对应于群的个数和状态. 期望最大化(Expectation maximum, EM)算法和马尔科夫链蒙特卡洛(Markov chain Monte Carlo, MCMC)算法分别被用于估计混合模型的参数. 混合模型的元素个数可通过删除、合并及分裂算法得到. 100次蒙特卡洛(Monte Carlo, MC)仿真实验表明该算法可有效跟踪部分可分辨的群目标. 相比EM算法, MCMC算法能够更好地提取群的个数和状态, 但它的计算量要大于EM算法.     

Gaussian mixture models with covariances or precisions in shared multiple subspaces

We introduce a class of Gaussian mixture models (GMMs) in which the covariances or the precisions (inverse covariances) are restricted to lie in subspaces spanned by rank-one symmetric matrices. The rank-one basis are shared between the Gaussians according to a sharing structure. We describe an algorithm for estimating the parameters of the GMM in a maximum likelihood framework given a sharing structure. We employ these models for modeling the observations in the hidden-states of a hidden Markov model based speech recognition system. We show that this class of models provide improvement in accuracy and computational efficiency over well-known covariance modeling techniques such as classical factor analysis, shared factor analysis and maximum likelihood linear transformation based models which are special instances of this class of models. We also investigate different sharing mechanisms. We show that for the same number of parameters, modeling precisions leads to better performance when compared to modeling covariances. Modeling precisions also gives a distinct advantage in computational and memory requirements.     

基于EM算法的混合模型医学图像分割

医学图像分割中的期望最大化(EM)算法在求解混合模型参数时存在局限性。为此,提出一种模糊约束的混合模型图像分割算法。该算法以像素的独立性假设为前提,在采用EM算法对模型参数进行求解的过程中,通过模糊集合论方法,引入像素空间信息。实验结果表明,该算法没有引入新的模型参数,能够保持独立混合模型的简单性,且具有自动模型选择能力,可以获得较理想的分割结果。     

A constrained EM algorithm for principal component analysis

We propose a constrained EM algorithm for principal component analysis (PCA) using a coupled probability model derived from single-standard factor analysis models with isotropic noise structure. The single probabilistic PCA, especially for the case where there is no noise, can find only a vector set that is a linear superposition of principal components and requires postprocessing, such as diagonalization of symmetric matrices. By contrast, the proposed algorithm finds the actual principal components, which are sorted in descending order of eigenvalue size and require no additional calculation or postprocessing. The method is easily applied to kernel PCA. It is also shown that the new EM algorithm is derived from a generalized least-squares formulation.     

Least-squares independence regression for non-linear causal inference under non-Gaussian noise

The discovery of non-linear causal relationship under additive non-Gaussian noise models has attracted considerable attention recently because of their high flexibility. In this paper, we propose a novel causal inference algorithm called least-squares independence regression (LSIR). LSIR learns the additive noise model through the minimization of an estimator of the squared-loss mutual information between inputs and residuals. A notable advantage of LSIR is that tuning parameters such as the kernel width and the regularization parameter can be naturally optimized by cross-validation, allowing us to avoid overfitting in a data-dependent fashion. Through experiments with real-world datasets, we show that LSIR compares favorably with a state-of-the-art causal inference method.     

A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning

The traditional Kalman filter can be viewed as a recursive stochastic algorithm that approximates an unknown function via a linear combination of prespecified basis functions given a sequence of noisy samples. In this paper, we generalize the algorithm to one that approximates the fixed point of an operator that is known to be a Euclidean norm contraction. Instead of noisy samples of the desired fixed point, the algorithm updates parameters based on noisy samples of functions generated by application of the operator, in the spirit of Robbins–Monro stochastic approximation. The algorithm is motivated by temporal-difference learning, and our developments lead to a possibly more efficient variant of temporal-difference learning. We establish convergence of the algorithm and explore efficiency gains through computational experiments involving optimal stopping and queueing problems. This research was supported in part by NSF CAREER Grant ECS-9985229, and by the ONR under Grant MURI N00014-00-1-0637.     

Speeding up the EM algorithm for mixture model-based segmentation of magnetic resonance images

Mixture models implemented via the expectation-maximization (EM) algorithm are being increasingly used in a wide range of problems in pattern recognition such as image segmentation. However, the EM algorithm requires considerable computational time in its application to huge data sets such as a three-dimensional magnetic resonance (MR) image of over 10 million voxels. Recently, it was shown that a sparse, incremental version of the EM algorithm could improve its rate of convergence. In this paper, we show how this modified EM algorithm can be speeded up further by adopting a multiresolution kd-tree structure in performing the E-step. The proposed algorithm outperforms some other variants of the EM algorithm for segmenting MR images of the human brain.     

Accelerating EM for Large Databases

The EM algorithm is a popular method for parameter estimation in a variety of problems involving missing data. However, the EM algorithm often requires significant computational resources and has been dismissed as impractical for large databases. We present two approaches that significantly reduce the computational cost of applying the EM algorithm to databases with a large number of cases, including databases with large dimensionality. Both approaches are based on partial E-steps for which we can use the results of Neal and Hinton (In Jordan, M. (Ed.), Learning in Graphical Models, pp. 355–371. The Netherlands: Kluwer Academic Publishers) to obtain the standard convergence guarantees of EM. The first approach is a version of the incremental EM algorithm, described in Neal and Hinton (1998), which cycles through data cases in blocks. The number of cases in each block dramatically effects the efficiency of the algorithm. We provide amethod for selecting a near optimal block size. The second approach, which we call lazy EM, will, at scheduled iterations, evaluate the significance of each data case and then proceed for several iterations actively using only the significant cases. We demonstrate that both methods can significantly reduce computational costs through their application to high-dimensional real-world and synthetic mixture modeling problems for large databases.     

Probabilistic sequence clustering with spectral learning

In this paper, we derive two novel learning algorithms for time series clustering; namely for learning mixtures of Markov Models and mixtures of Hidden Markov Models. Mixture models are special latent variable models that require the usage of local search heuristics such as Expectation Maximization (EM) algorithm, that can only provide locally optimal solutions. In contrast, we make use of the spectral learning algorithms, recently popularized in the machine learning community. Under mild assumptions, spectral learning algorithms are able to estimate the parameters in latent variable models by solving systems of equations via eigendecompositions of matrices or tensors of observable moments. As such, spectral methods can be viewed as an instance of the method of moments for parameter estimation, an alternative to maximum likelihood. The popularity stems from the fact that these methods provide a computationally cheap and local optima free alternative to EM. We conduct classification experiments on human action sequences extracted from videos, clustering experiments on motion capture data and network traffic data to illustrate the viability of our approach. We conclude that the spectral methods are a practical and useful alternative in terms of computational effort and solution quality to standard iterative techniques such as EM in several sequence clustering applications.     

并行的贝叶斯网络参数学习算法

针对大样本条件下EM算法学习贝叶斯网络参数的计算问题,提出一种并行EM算法(Parallel EM,PL-EM)提高大样本条件下复杂贝叶斯网络参数学习的速度.PL-EM算法在E步并行计算隐变量的后验概率和期望充分统计因子;在M步,利用贝叶斯网络的条件独立性和完整数据集下的似然函数可分解性,并行计算各个局部似然函数.实验结果表明PL-EM为解决大样本条件下贝叶斯网络参数学习提供了一种有效的方法.     

Online Noise Estimation Using Stochastic-Gain HMM for Speech Enhancement

We propose a noise estimation algorithm for single-channel noise suppression in dynamic noisy environments. A stochastic-gain hidden Markov model (SG-HMM) is used to model the statistics of nonstationary noise with time-varying energy. The noise model is adaptive and the model parameters are estimated online from noisy observations using a recursive estimation algorithm. The parameter estimation is derived for the maximum-likelihood criterion and the algorithm is based on the recursive expectation maximization (EM) framework. The proposed method facilitates continuous adaptation to changes of both noise spectral shapes and noise energy levels, e.g., due to movement of the noise source. Using the estimated noise model, we also develop an estimator of the noise power spectral density (PSD) based on recursive averaging of estimated noise sample spectra. We demonstrate that the proposed scheme achieves more accurate estimates of the noise model and noise PSD, and as part of a speech enhancement system facilitates a lower level of residual noise.     

Real-time motion segmentation of sparse feature points at any speed.

We present a real-time incremental approach to motion segmentation operating on sparse feature points. In contrast to previous work, the algorithm allows for a variable number of image frames to affect the segmentation process, thus enabling an arbitrary number of objects traveling at different relative speeds to be detected. Feature points are detected and tracked throughout an image sequence, and the features are grouped using a spatially constrained expectation-maximization (EM) algorithm that models the interactions between neighboring features using the Markov assumption. The primary parameter used by the algorithm is the amount of evidence that must accumulate before features are grouped. A statistical goodness-of-fit test monitors the change in the motion parameters of a group over time in order to automatically update the reference frame. Experimental results on a number of challenging image sequences demonstrate the effectiveness and computational efficiency of the technique.