基于粒子滤波的非线性系统静态参数估计方法*

针对基于滤波方法的最大似然参数估计步长序列过于单一,算法收敛缓慢并很容易收敛于局部最优解的问题,提出了基于似然权值的在线EM参数估计算法(LWOEM)。通过粒子滤波方法实时估计系统的状态值变化,结合最大似然方法计算静态参数的点估计,然后通过计算更新参数的似然值来动态更新步长序列.与在线EM参数估计算法(OEM)的实验结果比较,表明该算法具有更好的适应性和收敛效果。     

Asymptotic convergence rate of the EM algorithm for gaussian mixtures

It is well known that the convergence rate of the expectation-maximization (EM) algorithm can be faster than those of convention first-order iterative algorithms when the overlap in the given mixture is small. But this argument has not been mathematically proved yet. This article studies this problem asymptotically in the setting of gaussian mixtures under the theoretical framework of Xu and Jordan (1996). It has been proved that the asymptotic convergence rate of the EM algorithm for gaussian mixtures locally around the true solution Theta* is o(e(0. 5-epsilon)(Theta*)), where epsilon > 0 is an arbitrarily small number, o(x) means that it is a higher-order infinitesimal as x --> 0, and e(Theta*) is a measure of the average overlap of gaussians in the mixture. In other words, the large sample local convergence rate for the EM algorithm tends to be asymptotically superlinear when e(Theta*) tends to zero.     

Singularity and Slow Convergence of the EM algorithm for Gaussian Mixtures

Singularities in the parameter spaces of hierarchical learning machines are known to be a main cause of slow convergence of gradient descent learning. The EM algorithm, which is another learning algorithm giving a maximum likelihood estimator, is also suffering from its slow convergence, which often appears when the component overlap is large. We analyze the dynamics of the EM algorithm for Gaussian mixtures around singularities and show that there exists a slow manifold caused by a singular structure, which is closely related to the slow convergence of the EM algorithm. We also conduct numerical simulations to confirm the theoretical analysis. Through the simulations, we compare the dynamics of the EM algorithm with that of the gradient descent algorithm, and show that their slow dynamics are caused by the same singular structure, and thus they have the same behaviors around singularities.     

A review of the Expectation Maximization algorithm in data-driven process identification

The Expectation Maximization (EM) algorithm has been widely used for parameter estimation in data-driven process identification. EM is an algorithm for maximum likelihood estimation of parameters and ensures convergence of the likelihood function. In presence of missing variables and in ill conditioned problems, EM algorithm greatly assists the design of more robust identification algorithms. Such situations frequently occur in industrial environments. Missing observations due to sensor malfunctions, multiple process operating conditions and unknown time delay information are some of the examples that can resort to the EM algorithm. In this article, a review on applications of the EM algorithm to address such issues is provided. Future applications of EM algorithm as well as some open problems are also provided.     

基于分裂EM算法的GMM参数估计

期望最大化(Expectation Maximization,EM)算法是一种求参数极大似然估计的迭代算法,常用来估计混合密度分布模型的参数。EM算法的主要问题是参数初始化依赖于先验知识且在迭代过程中容易收敛到局部极大值。提出一种新的基于分裂EM算法的GMM参数估计算法,该方法从一个确定的单高斯分布开始,在EM优化过程中逐渐分裂并估计混合分布的参数,解决了参数迭代收敛到局部极值问题。大量的实验表明,与现有的其他参数估计算法相比,算法具有较好的运算效率和估算准确性。     

Adaptive control of continuous-time linear stochastic systems

An adaptive control problem for linear, continuous-time stochastic systems is described and solved in this paper. A solution of the adaptive control problem means that the family of maximum likelihood estimators is shown to be strongly consistent and the average costs are shown to converge to the optimal average costs. The unknown parameters in the model appear affinely in the drift term of the stochastic differential equation. The assumptions that are made for the solution are natural and verifiable. A recursive equation is given for the maximum likelihood estimates. This research was partially supported by NSF Grants ECS-8403286-A01 and ECS-8718026.     

TRUST-TECH-based expectation maximization for learning finite mixture models

In spite of the initialization problem, the Expectation-Maximization (EM) algorithm is widely used for estimating the parameters of finite mixture models. Most popular model-based clustering techniques might yield poor clusters if the parameters are not initialized properly. To reduce the sensitivity of initial points, a novel algorithm for learning mixture models from multivariate data is introduced in this paper. The proposed algorithm takes advantage of TRUST-TECH (TRansformation Under STability-reTaining Equilibra CHaracterization) to compute neighborhood local maxima on likelihood surface using stability regions. Basically, our method coalesces the advantages of the traditional EM with that of the dynamic and geometric characteristics of the stability regions of the corresponding nonlinear dynamical system of the log-likelihood function. Two phases namely, the EM phase and the stability region phase, are repeated alternatively in the parameter space to achieve improvements in the maximum likelihood. The EM phase obtains the local maximum of the likelihood function and the stability region phase helps to escape out of the local maximum by moving towards the neighboring stability regions. The algorithm has been tested on both synthetic and real datasets and the improvements in the performance compared to other approaches are demonstrated. The robustness with respect to initialization is also illustrated experimentally.     

A Semi-supervised Learning Algorithm on Gaussian Mixture with Automatic Model Selection

In Gaussian mixture modeling, it is crucial to select the number of Gaussians for a sample set, which becomes much more difficult when the overlap in the mixture is larger. Under regularization theory, we aim to solve this problem using a semi-supervised learning algorithm through incorporating pairwise constraints into entropy regularized likelihood (ERL) learning which can make automatic model selection for Gaussian mixture. The simulation experiments further demonstrate that the presented semi-supervised learning algorithm (i.e., the constrained ERL learning algorithm) can automatically detect the number of Gaussians with a good parameter estimation, even when two or more actual Gaussians in the mixture are overlapped at a high degree. Moreover, the constrained ERL learning algorithm leads to some promising results when applied to iris data classification and image database categorization.     

一种半监督机器学习的EM算法改进方法

EM(Expectation Maximization)算法是含有隐变量(latent variable)的概率参数模型最大似然估计、极大后验概率估计最有效的算法,但很容易进入局部最优现象,对此提出基于半监督机器学习机制的EM算法.本文方法是在最大似然函数中加入惩罚最小二乘因子,同时引入非负约束作为先验信息,结合半监督机器学习方法,将EM算法改进转化为最小化求解问题,再采用最大似然方法求解EM模型,有效估计了混合矩阵和高斯混合模型参数,实现EM算法的改进.仿真结果表明,该方法能够很好地解决了EM算法容易局部最优化问题.     

Blind equalization using parallel Bayesian decision feedback equalizer

The purpose of this paper is to propose a new method for blind equalization using parallel Bayesian decision feedback equalizer (DFE). Blind equalization based on decision-directed algorithm, including the previous proposed Chen’s blind Bayesian DFE, cannot give the correct convergence without the suitable initialization corresponding to the small inter-symbol interference. How to find the suitable initialization becomes the key for the correct convergence. Here, the “start” vector with several states is used to obtain several channel estimates which are the initial channel estimates in proposed method. In these initial channel estimates, the best one which has converged toward the correct result in some degree must exist. The decision-directed algorithm for parallel blind Bayesian DFE is purchased from these initial channel estimates respectively. Evaluating the Bayesian likelihood which is defined as the accumulation of the natural logarithm of the Bayesian decision variable, the correct channel estimates corresponding to the maximum Bayesian likelihood can be found. Compared with Chen’s blind Bayesian DFE, the proposed method presents better convergence performance with less computational complexity. Furthermore, the proposed algorithm works satisfactorily even for channel with severe ISI and in-band spectral null, while Chen’s blind Bayesian DFE fails.     

Robust Gaussian process modeling using EM algorithm

Gaussian process (GP) regression is a fully probabilistic method for performing non-linear regression. In a Bayesian framework, regression models can be made robust by using heavy-tailed distributions instead of using normal distribution for modeling noise. This work focuses on estimation of parameters for robust GP regression. In literature, these are learned by maximizing the approximate marginal likelihood of data. However, gradient-based optimization algorithms which are used for this purpose can be unstable or may require tuning. In this work, an EM algorithm based approach is derived and implemented to infer the parameters. The pros and cons of the two approaches are analyzed. The advantage of EM algorithm lies in its ease of implementation and theoretical guarantees of numerical stability and convergence while its prediction performance is still comparable to gradient-based approaches. In some cases EM algorithm may be slow to converge. To circumvent this issue a faster EM based approach known as Expectation Conjugate Gradient (ECG) is implemented on robust GP regression. Finally, the proposed EM approach to robust GP regression is validated using an industrial data set.     

Parameter estimation of two-level nonlinear mixed effects models using first order conditional linearization and the EM algorithm

Multi-level nonlinear mixed effects (ML-NLME) models have received a great deal of attention in recent years because of the flexibility they offer in handling the repeated-measures data arising from various disciplines. In this study, we propose both maximum likelihood and restricted maximum likelihood estimations of ML-NLME models with two-level random effects, using first order conditional expansion (FOCE) and the expectation–maximization (EM) algorithm. The FOCE–EM algorithm was compared with the most popular Lindstrom and Bates (LB) method in terms of computational and statistical properties. Basal area growth series data measured from Chinese fir (Cunninghamia lanceolata) experimental stands and simulated data were used for evaluation. The FOCE–EM and LB algorithms given the same parameter estimates and fit statistics for models that converged by both. However, FOCE–EM converged for all the models, while LB did not, especially for the models in which two-level random effects are simultaneously considered in several base parameters to account for between-group variation. We recommend the use of FOCE–EM in ML-NLME models, particularly when convergence is a concern in model selection.     

基于EM算法的混合模型的参数估计

介绍了极大似然参数估计，然后介绍了混合模型极大似然参数估计的EM算法实现，最后利用计算机仿真实验验证了此算法的有效性和收敛性．     

一种快速放射CT重建算法的研究

ＥＭ算法用于求解重建问题，具有一些非常好的特性，几十年来一直为人们所重视。ＥＭ算法也存在着缺点，例如收敛速度较慢，使它的应用受到一定的限制。为此，人们提出了很多加快收敛速度的方法，取得了不少有价值的结果。     

退火期望最大化算法A-EM

使用EM算法训练随机多层前馈网具有低开销、易于实现和全局收敛的特点,在EM算法的基础上提出了一种训练随机多层前馈网络的新方法AEM.AEM算法利用热力学系统的最大熵原理计算网络中隐变量的条件概率,借鉴退火过程,引入温度参数,减小了初始参数值对最终结果的影响.该算法既保持了原EM算法的优点,又有利于训练结果收敛到全局极小.从数学角度证明了该算法的收敛性,同时,实验也证明了该算法的正确性和有效性.     

基于噪音受益的快速图像分割算法

图像分割是指将一幅图像分解为若干互不交迭的区域的集合。当用已有的改进高斯混合模型于图像分割时,如何加快其分割过程是一个有研究意义的课题。基于最新的噪音受益EM算法,通过人工加噪来加快已有的改进高斯混合模型的收敛速度,从而达到加快图像分割的目的。当添加的噪声满足噪音受益EM定理时,加性噪声加快了EM算法收敛到局部最大值的平均收敛速度。改进的高斯混合模型是EM算法的特例,因此,噪音受益EM定理同样适用于改进的高斯混合模型。实验表明,提出的算法进行图像分割时,其收敛速度明显加快,时间复杂度明显变小。     

A classification EM algorithm for binned data

A real-time flaw diagnosis application for pressurized containers using acoustic emissions is described. The pressurized containers used are cylindrical tanks containing fluids under pressure. The surface of the pressurized containers is divided into bins, and the number of acoustic signals emanating from each bin is counted. Spatial clustering of high density bins using mixture models is used to detect flaws. A dedicated EM algorithm can be derived to select the mixture parameters, but this is a greedy algorithm since it requires the numerical computation of integrals and may converge only slowly. To deal with this problem, a classification version of the EM (CEM) algorithm is defined, and using synthetic and real data sets, the proposed algorithm is compared to the CEM algorithm applied to classical data. The two approaches generate comparable solutions in terms of the resulting partition if the histogram is sufficiently accurate, but the algorithm designed for binned data becomes faster when the number of available observations is large enough.     

Maximum likelihood stochastic gradient estimation for Hammerstein systems with colored noise based on the key term separation technique

This paper considers the identification problems of Hammerstein finite impulse response moving average (FIR-MA) systems using the maximum likelihood principle and stochastic gradient method based on the key term separation technique. In order to improve the convergence rate, a maximum likelihood multi-innovation stochastic gradient algorithm is presented. The simulation results show that the proposed algorithms can effectively estimate the parameters of the Hammerstein FIR-MA systems.     

A new lifetime distribution

A new two-parameter distribution with decreasing failure rate is introduced. Various properties of the introduced distribution are discussed. The EM algorithm is used to determine the maximum likelihood estimates and the asymptotic variances and covariance of these estimates are obtained. Simulation studies are performed in order to assess the accuracy of the approximation of the variances and covariance of the maximum likelihood estimates and investigate the convergence of the proposed EM scheme. Illustrative examples based on real data are also given.     

期望最大算法及其应用（2）

EM算法是实现极大似然估计的一种有效方法,主要用于非完全数据的参数估计。文章的第一部分已经详细介绍了算法的基本原理,这部分内容着重介绍算法的各种应用,特别是高斯混合模型、隐马尔科夫模型和因子分析中的参数估计。