Maximum likelihood parameter estimation algorithm for controlled autoregressive autoregressive models

A maximum likelihood parameter estimation algorithm is derived for controlled autoregressive autoregressive (CARAR) models based on the maximum likelihood principle. In this derivation, we use an estimated noise transfer function to filter the input–output data. The simulation results show that the proposed estimation algorithm can effectively estimate the parameters of such class of CARAR systems and give more accurate parameter estimates than the recursive generalized least-squares algorithm.     

高速列车非线性模型的极大似然辨识

提出高速列车非线性模型的极大似然(Maximum likelihood, ML)辨识方法,适合于高速列车在非高斯噪声干扰下的非线性模型的参数估计.首先,构建了描述高速列车单质点力学行为的随机离散非线性状态空间模型,并将高速列车参数的极大似然(ML)估计问题转化为期望极大(Expectation maximization, EM)的优化问题; 然后,给出高速列车状态估计的粒子滤波器和粒子平滑器的设计方法,据此构造列车的条件数学期望,并给出最大化该数学期望的梯度搜索方法,进而得到列车参数的辨识算法,分析了算法的收敛速度; 最后,进行了高速列车阻力系数估计的数值对比实验. 结果表明, 所提出的辨识方法的有效性.     

Reliability estimation in Lindley distribution with progressively type II right censored sample

In this paper we discuss one parameter Lindley distribution. It is suggested that it may serve as a useful reliability model. The model properties and reliability measures are derived and studied in detail. For the estimation purposes of the parameter and other reliability characteristics maximum likelihood and Bayes approaches are used. Interval estimation and coverage probability for the parameter are obtained based on maximum likelihood estimation. Monte Carlo simulation study is conducted to compare the performance of the various estimates developed. In view of cost and time constraints, progressively Type II censored sample data are used in estimation. A real data example is given for illustration.     

二值时间序列的一种阈值检测方法

二值时间序列的检测问题存在于具有跳变输入的Ｋａｌｍａｎ滤波及最优平滑等实际应用中，本文基于系统平滑估值的频域特性和极大似然原理，给出一种阈值检测方法，算法可用一次最优平滑实现，仿真实验验证了方法的有效性。     

Constructing maximum likelihood estimates for statistically uncertain linear systems

We consider the parameters estimation problem for a statistically uncertain linear model, i.e., a model whose observations contain both random perturbations with known distributions and uncertain perturbations for which we only know the domain of their possible values. To solve this problem, we use an approach related to the maximum likelihood method for statistically uncertain systems. We show that as the variances of random perturbations tend to zero, maximum likelihood estimates converge to the information set of the system without random perturbations.     

Identification of non-linear stochastic systems by state dependent parameter estimation

The paper outlines how improved estimates of time variable parameters in models of stochastic dynamic systems can be obtained using recursive filtering and fixed interval smoothing techniques, with the associated hyper-parameters optimized by maximum likelihood based on prediction error decomposition. It then shows how, by exploiting special data re-ordering and back-fitting procedures, similar recursive parameter estimation techniques can be utilized to estimate much more rapid State Dependent Parameter (SDP) variations. In this manner, it is possible to identify and estimate a widely applicable class of nonlinear stochastic systems, as illustrated by several examples that include simulated and real data from chaotic processes. Finally, the paper points out that such SDP models can form the basis for new methods of signal processing, automatic control and state estimation for nonlinear stochastic systems.     

Quasi-Monte Carlo estimation in generalized linear mixed models

Generalized linear mixed models (GLMMs) are useful for modelling longitudinal and clustered data, but parameter estimation is very challenging because the likelihood may involve high-dimensional integrals that are analytically intractable. Gauss-Hermite quadrature (GHQ) approximation can be applied but is only suitable for low-dimensional random effects. Based on the Quasi-Monte Carlo (QMC) approximation, a heuristic approach is proposed to calculate the maximum likelihood estimates of parameters in the GLMM. The QMC points scattered uniformly on the high-dimensional integration domain are generated to replace the GHQ nodes. Compared to the GHQ approximation, the proposed method has many advantages such as its affordable computation, good approximation and fast convergence. Comparisons to the penalized quasi-likelihood estimation and Gibbs sampling are made using a real dataset and a simulation study. The real dataset is the salamander mating dataset whose modelling involves six 20-dimensional intractable integrals in the likelihood.     

Quasi-Monte Carlo estimation in generalized linear mixed models

Generalized linear mixed models (GLMMs) are useful for modelling longitudinal and clustered data, but parameter estimation is very challenging because the likelihood may involve high-dimensional integrals that are analytically intractable. Gauss–Hermite quadrature (GHQ) approximation can be applied but is only suitable for low-dimensional random effects. Based on the Quasi-Monte Carlo (QMC) approximation, a heuristic approach is proposed to calculate the maximum likelihood estimates of parameters in the GLMM. The QMC points scattered uniformly on the high-dimensional integration domain are generated to replace the GHQ nodes. Compared to the GHQ approximation, the proposed method has many advantages such as its affordable computation, good approximation and fast convergence. Comparisons to the penalized quasi-likelihood estimation and Gibbs sampling are made using a real dataset and a simulation study. The real dataset is the salamander mating dataset whose modelling involves six 20-dimensional intractable integrals in the likelihood.     

A comparison of simultaneous state and parameter estimation schemes for a continuous fermentor reactor

This article proposes a maximum likelihood algorithm for simultaneous estimation of state and parameter values in nonlinear stochastic state-space models. The proposed algorithm uses a combination of expectation maximization, nonlinear filtering and smoothing algorithms. The algorithm is tested with three popular techniques for filtering namely particle filter (PF), unscented Kalman filter (UKF) and extended Kalman filter (EKF). It is shown that the proposed algorithm when used in conjunction with UKF is computationally more efficient and provides better estimates. An online recursive algorithm based on nonlinear filtering theory is also derived and is shown to perform equally well with UKF and ensemble Kalman filter (EnKF) algorithms. A continuous fermentation reactor is used to illustrate the efficacy of batch and online versions of the proposed algorithms.     

Consistency of modified LS estimation method for identifying 2-Dnoncausal SAR model parameters

Least squares (LS) and maximum likelihood (ML) are the two main methods for parameter estimation of two-dimensional (2D) noncausal simultaneous autoregressive (SAR) models. ML is asymptotically consistent and unbiased but computationally unattractive. On the other hand, conventional LS is computationally efficient but does not produce accurate parameter estimates for noncausal models. Recently, Zhao-Yu (1993) proposed a modified LS estimation method and was shown to be unbiased. In this paper we prove that, under certain assumptions, the method introduced by Zhao-Yu is also consistent     

Multivariable process identification for mpc: the asymptotic method and its applications

In this work we will introduce the asymptotic method (ASYM) of identification and provide two case studies. The ASYM was developed for multivariable process identification for model based control. The method calculates time domain parametric models using frequency domain criterion. Fundamental problems, such as test signal design for control, model order/structure selection, parameter estimation and model error quantification, are solved in a systematic manner. The method can supply not only input/output model and unmeasured disturbance model which are asymptotic maximum likelihood estimates, but also the upper bound matrix for the model errors that can be used for model validation and robustness analysis. To demonstrate the use of the method for model predictive control (MPC), the identification of a Shell benchmark process (a simulated distillation column) and an industrial application to a crude unit atmospheric tower will be presented.     

Penalized spline estimation for functional coefficient regression models

The functional coefficient regression models assume that the regression coefficients vary with some “threshold” variable, providing appreciable flexibility in capturing the underlying dynamics in data and avoiding the so-called “curse of dimensionality” in multivariate nonparametric estimation. We first investigate the estimation, inference, and forecasting for the functional coefficient regression models with dependent observations via penalized splines. The P-spline approach, as a direct ridge regression shrinkage type global smoothing method, is computationally efficient and stable. With established fixed-knot asymptotics, inference is readily available. Exact inference can be obtained for fixed smoothing parameter λ, which is most appealing for finite samples. Our penalized spline approach gives an explicit model expression, which also enables multi-step-ahead forecasting via simulations. Furthermore, we examine different methods of choosing the important smoothing parameter λ: modified multi-fold cross-validation (MCV), generalized cross-validation (GCV), and an extension of empirical bias bandwidth selection (EBBS) to P-splines. In addition, we implement smoothing parameter selection using mixed model framework through restricted maximum likelihood (REML) for P-spline functional coefficient regression models with independent observations. The P-spline approach also easily allows different smoothness for different functional coefficients, which is enabled by assigning different penalty λ accordingly. We demonstrate the proposed approach by both simulation examples and a real data application.     

SAS and SPLUS programs to perform Cox regression without convergence problems

When analyzing survival data, the parameter estimates and consequently the relative risk estimates of a Cox model sometimes do not converge to finite values. This phenomenon is due to special conditions in a data set and is known as 'monotone likelihood'. Statistical software packages for Cox regression using the maximum likelihood method cannot appropriately deal with this problem. A new procedure to solve the problem has been proposed by G. Heinze, M. Schemper, A solution to the problem of monotone likelihood in Cox regression, Biometrics 57 (2001). It has been shown that unlike the standard maximum likelihood method, this method always leads to finite parameter estimates. We developed a SAS macro and an SPLUS library to make this method available from within one of these widely used statistical software packages. Our programs are also capable of performing interval estimation based on profile penalized log likelihood (PPL) and of plotting the PPL function as was suggested by G. Heinze, M. Schemper, A solution to the problem of monotone likelihood in Cox regression, Biometrics 57 (2001).     

A computational strategy for doubly smoothed MLE exemplified in the normal mixture model

A typical problem for the parameter estimation in normal mixture models is an unbounded likelihood and the presence of many spurious local maxima. To resolve this problem, we apply the doubly smoothed maximum likelihood estimator (DS-MLE) proposed by Seo and Lindsay (in preparation). We discuss the computational issues of the DS-MLE and propose a simulation-based DS-MLE using Monte Carlo methods as a general computational tool. Simulation results show that the DS-MLE is virtually consistent for any bandwidth choice. Moreover, the parameter estimates in the DS-MLE are quite robust to the choice of bandwidths, as the theory indicates. A new method for the bandwidth selection is also proposed.     

Parameter estimation in multivariate stochastic difference equations

We will review the principal methods of estimation of parameters in multivariate autoregressive moving average equations which have additional observable input terms in them and present some new methods of estimation as well. We begin with the conditions for the estimability of the parameters. In addition to the usual method of system representation, the canonical form I, we will present two new representations of the system equation, the so-called canonical forms II and III which are convenient for parameter estimation. We will mention, in some detail, the various methods of estimation like the various least-squares methods, the maximum likelihood methods, etc., and discuss them regarding their relative accuracy of the estimate and the corresponding computational complexity. We will introduce a new class of estimates, the so-called limited information estimates which utilizes the canonical forms II and III. The accuracy of these estimates is close to that of maximum likelihood, but their computation time is only a fraction of the computation time for the usual maximum likelihood estimates. We will present a few numerical examples to illustrate the various methods.     

On the use of constraints in least squares estimation and control

In this article, we examine the effect of constraints on estimation and control methods based on quadratic penalty functions. We begin with estimation theory and analyze how constraints alter the statistical properties of the least squares estimates. It is shown that constraints can be used to formulate maximum likelihood (MLE) and maximum a posteriori (MAP) estimators for a variety of unimodal distributions. This provides greater flexibility over the assumption of normality inherent in the MLE and MAP interpretation of traditional least squares. We discuss how these ideas apply to state space models of dynamic systems. Possible applications for controllers that handle constraints are also discussed. A parameter estimation example is given to demonstrate the potential for improved performance over unconstrained least squares.     

An adjustment for edge effects using an augmented neighborhood model in the spatial auto-logistic model

Parameter estimation in the spatial auto-regressive models has difficulty due to the edge sites which have unobserved neighborhood sites. Some ad hoc remedies suggested in the literature are the free boundary condition, the toroidal boundary condition or estimation using only internal data sites. However, parameter estimates are often sensitive to assumptions on the unobserved neighborhood sites and all the above assumptions have some apparent shortcomings such as systematic bias or inflated variance. In this paper, we propose a new way to incorporate the edge sites by introducing an augmented random neighborhood, denoted by the augmented neighborhood model, which represents the entire external field. To estimate the model, we derive the EM procedures for the maximum pseudo-likelihood estimator and the maximum likelihood estimator. Several simulation studies show that the random external field provides better performance of the maximum pseudo-likelihood estimator and the maximum likelihood estimator than other assumptions on the edge sites. As an example, we apply the random external field to modeling the distribution of Plantago lanceolata in Kansas.     

A Gaussian pseudolikelihood approach for quantile regression with repeated measurements

To enhance the efficiency of regression parameter estimation by modeling the correlation structure of correlated binary error terms in quantile regression with repeated measurements, we propose a Gaussian pseudolikelihood approach for estimating correlation parameters and selecting the most appropriate working correlation matrix simultaneously. The induced smoothing method is applied to estimate the covariance of the regression parameter estimates, which can bypass density estimation of the errors. Extensive numerical studies indicate that the proposed method performs well in selecting an accurate correlation structure and improving regression parameter estimation efficiency. The proposed method is further illustrated by analyzing a dental dataset.     

Stepwise variable selection and maximum likelihood estimation of smoothing factors of kernel functions for nonparametric discriminant functions evaluated by different criteria

Kernel functions are used to estimate the probability density functions of variables for nonparametric discriminant analysis. In connection with stepwise variable identification a stepwise maximum likelihood estimation procedure for the estimation of smoothing factors of the kernel functions is developed. This procedure allows a step-by-step estimation of smoothing factors for every variable which is considered to be added to the model or which is examined to substitute a variable in a model. Different criteria for model evaluation in stepwise discriminant analysis are discussed. Beside criteria, like distance and dependence functions and the error and nonerror rate, a criterion which considers the ratio of probability densities of different classes at point x is proposed for stepwise variable identification. An application of the procedures described in this study to a medical decision problem shows the importance of stepwise parameter estimation of kernel functions for nonparametric discriminant analysis and the role of different model evaluation criteria for the selection of the best subset of variables.     

Spectral estimation of a structural thin-plate smoothing model

A nonstationary structural spatial model that explicitly sets the data to evolve across a rectangular lattice constrained by second-order smoothing restrictions is presented. The model exemplifies the concept of model-based spatial smoothing and, in particular, it provides a rationale for the popular discrete thin-plate smoothing method. It is further shown how to use a frequency-domain approach to estimate the spatial model via maximum likelihood. In essence, the approach allows both dimensions to be treated separately from each other so that the computational burden for the estimation of two-dimensional models is dramatically reduced both in terms of the computing time and the memory required. Besides, this spectral approach allows straightforward construction of analytic derivatives and an expression for the asymptotic variance of the estimated smoothing parameter is derived with which to construct confidence intervals. Some numerical Monte–Carlo evidence and one example illustrate the results given.