Nonparametric Stein-type shrinkage covariance matrix estimators in high-dimensional settings

Estimating a covariance matrix is an important task in applications where the number of variables is larger than the number of observations. Shrinkage approaches for estimating a high-dimensional covariance matrix are often employed to circumvent the limitations of the sample covariance matrix. A new family of nonparametric Stein-type shrinkage covariance estimators is proposed whose members are written as a convex linear combination of the sample covariance matrix and of a predefined invertible target matrix. Under the Frobenius norm criterion, the optimal shrinkage intensity that defines the best convex linear combination depends on the unobserved covariance matrix and it must be estimated from the data. A simple but effective estimation process that produces nonparametric and consistent estimators of the optimal shrinkage intensity for three popular target matrices is introduced. In simulations, the proposed Stein-type shrinkage covariance matrix estimator based on a scaled identity matrix appeared to be up to 80% more efficient than existing ones in extreme high-dimensional settings. A colon cancer dataset was analyzed to demonstrate the utility of the proposed estimators. A rule of thumb for adhoc selection among the three commonly used target matrices is recommended.     

Novel estimators of equivalent number of looks in polarimetric SAR imagery based on sub-matrices基于子矩阵的极化SAR图像等效视图数估计方法

The equivalent number of looks (ENL) is an important parameter in the multilook statistical model of polarimetric synthetic aperture radar (Pol-SAR). Recently, the maximum likelihood (ML) method was proposed and gave a good performance in the Gaussian model case by using the full covariance matrix instead of the intensity of Pol-SAR data, but it generated underestimates in the product model case. In this paper several novel ENL estimators are presented via certain cumulants of the log-determinant of the sub-matrices of the multilook polarimetric covariance matrix. The texture effect to the ENL estimates is eliminated, and the analytic estimators are derived. The estimators use the full covariance matrix and sub-matrices information, rather than the intensities of polarization channels. All the novel estimators are suitable for any texture model and thus provide more accurate results than many existing ones. Experiments using simulated data and real data are presented to evaluate the performance of different estimators. The results show that the second log-determinant moment (SLDM3)-based method is the best one among the novel estimators. At the same time this estimator has much less computational complexity. In addition, a novel distribution classification method is proposed by coloring the image via second- and third-order log-cumulants of the covariance matrix (MLC), which is helpful to assess the estimation result.     

Rank regression for analysis of clustered data: A natural induced smoothing approach

We consider rank regression for clustered data analysis and investigate the induced smoothing method for obtaining the asymptotic covariance matrices of the parameter estimators. We prove that the induced estimating functions are asymptotically unbiased and the resulting estimators are strongly consistent and asymptotically normal. The induced smoothing approach provides an effective way for obtaining asymptotic covariance matrices for between- and within-cluster estimators and for a combined estimator to take account of within-cluster correlations. We also carry out extensive simulation studies to assess the performance of different estimators. The proposed methodology is substantially much faster in computation and more stable in numerical results than the existing methods. We apply the proposed methodology to a dataset from a randomized clinical trial.     

Comparison of various estimators in simulated FGN

The Hurst parameter is the simplest numerical characteristic of self-similar long-range dependent stochastic processes. Such processes have been identified in many natural and man-made systems. In particular, since they were discovered in the Internet and other multimedia telecommunication networks a decade ago, they have been the subject of numerous investigations. Typical quantitative assessment of self-similarity and long-range dependency, begins with the estimation of the Hurst parameter H. There have been a number of techniques proposed for this. This paper reports results of a comparative analysis of the six most frequently used estimators of H. To set up a credible framework for this, the minimal acceptable sample size is first determined. The Hurst parameter estimators are then compared for bias and variance. Our experimental results have confirmed that the Abry–Veitch Daubechies Wavelet-Based (DWB) and the Whittle ML (Maximum Likelihood) estimators of H are the least biased. However, the latter has significantly smaller variance and can be applied to shorter data samples than the Abry–Veitch DWB estimator. On the other hand, the Abry–Veitch DWB estimator is computationally simpler and faster than the Whittle ML estimator.     

Development of moving window state and parameter estimators under maximum likelihood and Bayesian frameworks

Estimation of slowly varying model parameters/unmeasured disturbances is of paramount importance in process monitoring, fault diagnosis, model based advanced control and online optimization. The conventional approach to estimate drifting parameters is to artificially model them as a random walk process and estimate them simultaneously with the states. However, this may lead to a poorly conditioned problem, where the tuning of the random walk model becomes a non-trivial exercise. In this work, the moving window parameter estimator of Huang et al. [1] is recast as a moving window maximum likelihood (ML) estimator. The state can be estimated within the window using any recursive Bayesian estimator. It is assumed that, when the model parameters are perfectly known, the innovation sequence generated by the chosen Bayesian estimator is a Gaussian white noise process and is further used to construct a likelihood function that treats the model parameters as unknowns. This leads to a well conditioned problem where the only tuning parameter is the length of the moving window, which is much easier to select than selecting the covariance of the random walk model. The ML formulation is further modified to develop a maximum a posteriori (MAP) cost function by including arrival cost for the parameter. Efficacy of the proposed ML and MAP formulations has been demonstrated by conducting simulation studies and experimental evaluation. Analysis of the simulation and experimental results reveals that the proposed moving window ML and MAP estimators are capable of tracking the drifting parameters/unmeasured disturbances fairly accurately even when the measurements are available at multiple rates and with variable time delays.     

One-step minimum Hellinger distance estimation

It is well known now that the minimum Hellinger distance estimation approach introduced by Beran (Beran, R., 1977. Minimum Hellinger distance estimators for parametric models. Ann. Statist. 5, 445-463) produces estimators that achieve efficiency at the model density and simultaneously have excellent robustness properties. However, computational difficulties and algorithmic convergence problems associated with this method have hampered its application in practice, particularly when the method is applied to models with high-dimensional parameter spaces. A one-step minimum Hellinger distance (MHD) procedure is investigated in this paper to overcome computational drawbacks of the fully iterative MHD method. The idea is to start with an initial estimator, and then iterate the Newton-Raphson equation once related to the Hellinger distance. The resulting estimator can be considered a one-step MHD estimator. We show that the proposed one-step MHD estimator has the same asymptotic behavior as the MHD estimator, as long as the initial estimators are reasonably good. Furthermore, our theoretical and numerical studies also demonstrate that the proposed one-step MHD estimator also retains excellent robustness properties of the MHD estimators. A real data example is analyzed as well.     

Direct estimation of shape from texture

Witkin (1981) has proposed a maximum likelihood (ML) estimator of surface orientation based on the observed directional bias of projected texture elements. However, a drawback of this procedure is that the estimate is only defined indirectly in terms of a set of nonlinear equations. An alternative method is proposed, which allows an estimate of the surface orientation to be computed directly in a single step from certain simple statistics of the image data. We also show that this direct estimate allows Witkin's ML estimate to be computed to within 0.05° in only two or three iterative steps. The performance of the new estimator is demonstrated experimentally and compared to that of the ML estimator, using both synthetic data and real gray-level images     

Some practical aspects of covariance estimation

Vectorcardiogram (VCG) data often are analyzed using the Karhunen-Loéve expansion of the sample covariance matrix, S, as a method for discriminating between the VCG's of healthy pat unhealthy patients. The estimator, S, however, can be seriously effected by both atypical observations and the number of VCG's in the database relative to their dimension. In this paper it is shown that alternative robust estimators of the covariance matrix are appealing in analyzing VCG data when outliers are present in the sample. Also, it is demonstrated that sample sizes in such experiments should be greatly expanded in order to validate the asymptotic properties of S.     

On estimation of nonlinear functionals from discrete noisy measurements

The principal objective of this paper is to estimate a nonlinear functional of state vector (NFS) in dynamical system. The NFS represents a multivariate functional of state variables which carries useful information of a target system for control. The paper focuses on estimation of the NFS in linear continuous-discrete systems. The optimal nonlinear estimator based on the minimum mean square error approach is derived. The estimator depends on the Kalman estimate of a state vector and its error covariance. Some challenging computational aspects of the optimal nonlinear estimator are solved by usage of the unscented transformation for implementation of the nonlinear estimator. The special quadratic functional of state vector (QFS) is studied in detail. We derive effective matrix formulas for the optimal quadratic estimator and mean square error. The quadratic estimator has a simple closed-form calculation procedure and it is easy to implement in practice. The obtained results we demonstrate on theoretical and practical examples with different types of an nonlinear functionals. Comparison analysis of the optimal and suboptimal estimators is presented. The subsequent application of the proposed optimal nonlinear and quadratic estimators demonstrates their effectiveness.     

Box-Jenkins identification revisited—Part I: Theory

In classical time domain Box-Jenkins identification discrete-time plant and noise models are estimated using sampled input/output signals. The frequency content of the input/output samples covers uniformly the whole unit circle in a natural way, even in case of prefiltering. Recently, the classical time domain Box-Jenkins framework has been extended to frequency domain data captured in open loop. The proposed frequency domain maximum likelihood (ML) solution can handle (i) discrete-time models using data that only covers a part of the unit circle, and (ii) continuous-time models. Part I of this series of two papers (i) generalizes the frequency domain ML solution to the closed loop case, and (ii) proves the properties of the ML estimator under non-standard conditions. Contrary to the classical time domain case it is shown that the controller should be either known or estimated. The proposed ML estimators are applicable to frequency domain data as well as time domain data.     

Robust concentration graph model selection

Concentration graph models are an attractive tool to explore the conditional independence structure in a multivariate normal distribution. In applications, in absence of a priori knowledge, it is possible to select the graph underlying a set of data through an appropriate model selection procedure. The recently proposed procedure, SINful, is appealing but sensitive to outliers, as it utilizes the sample estimator of the covariance matrix. A method to make the SINful procedure robust with respect to the presence of outlying observations, is proposed. This is based on the minimum covariance determinant (MCD) estimator for the variance-covariance matrix. A simulation study shows the advantages of this method.     

Relaxed Lasso

The Lasso is an attractive regularisation method for high-dimensional regression. It combines variable selection with an efficient computational procedure. However, the rate of convergence of the Lasso is slow for some sparse high-dimensional data, where the number of predictor variables is growing fast with the number of observations. Moreover, many noise variables are selected if the estimator is chosen by cross-validation. It is shown that the contradicting demands of an efficient computational procedure and fast convergence rates of the ?₂-loss can be overcome by a two-stage procedure, termed the relaxed Lasso. For orthogonal designs, the relaxed Lasso provides a continuum of solutions that include both soft- and hard-thresholding of estimators. The relaxed Lasso solutions include all regular Lasso solutions and computation of all relaxed Lasso solutions is often identically expensive as computing all regular Lasso solutions. Theoretical and numerical results demonstrate that the relaxed Lasso produces sparser models with equal or lower prediction loss than the regular Lasso estimator for high-dimensional data.     

On convergence of covariancematrix of empirical Bayes hyper-parameter estimator

Regularized system identification has become the research frontier of system identification in the past decade. One related core subject is to study the convergence properties of various hyper-parameter estimators as the sample size goes to infinity. In this paper, we consider one commonly used hyper-parameter estimator, the empirical Bayes (EB). Its convergence in distribution has been studied, and the explicit expression of the covariance matrix of its limiting distribution has been given. However, what we are truly interested in are factors contained in the covariance matrix of the EB hyper-parameter estimator, and then, the convergence of its covariance matrix to that of its limiting distribution is required. In general, the convergence in distribution of a sequence of random variables does not necessarily guarantee the convergence of its covariance matrix. Thus, the derivation of such convergence is a necessary complement to our theoretical analysis about factors that influence the convergence properties of the EB hyper-parameter estimator. In this paper, we consider the regularized finite impulse response (FIR) model estimation with deterministic inputs, and show that the covariance matrix of the EB hyper-parameter estimator converges to that of its limiting distribution. Moreover, we run numerical simulations to demonstrate the efficacy of our theoretical results.     

Hyper least squares fitting of circles and ellipses

This work extends the circle fitting method of Rangarajan and Kanatani (2009) to accommodate ellipse fitting. Our method, which we call HyperLS, relies on algebraic distance minimization with a carefully chosen scale normalization. The normalization is derived using a rigorous error analysis of least squares (LS) estimators so that statistical bias is eliminated up to second order noise terms. Numerical evidence suggests that the proposed HyperLS estimator is far superior to the standard LS and is slightly better than the Taubin estimator. Although suboptimal in comparison to maximum likelihood (ML), our HyperLS does not require iterations. Hence, it does not suffer from convergence issues due to poor initialization, which is inherent in ML estimators. In this sense, the proposed HyperLS is a perfect candidate for initializing the ML iterations.     

Design-based approach to k-nearest neighbours technique for coupling field and remotely sensed data in forest surveys

The statistical properties of the k-NN estimators are investigated in a design-based framework, avoiding any assumption about the population under study. The issue of coupling remotely sensed digital imagery with data arising from forest inventories conducted using probabilistic sampling schemes is considered. General results are obtained for the k-NN estimator at the pixel level. When averages (or totals) of forest attributes for the whole study area or sub-areas are of interest, the use of the empirical difference estimator is proposed. The estimator is shown to be approximately unbiased with a variance admitting unbiased or conservative estimators. The performance of the empirical difference estimator is evaluated by an extensive simulation study performed on several populations whose dimensions and covariate values are taken from a real case study. Samples are selected from the populations by means of simple random sampling without replacement. Comparisons with the generalized regression estimator and Horvitz-Thompson estimators are also performed. An application to a local forest inventory on a test area of central Italy is considered.     

Absolute penalty and shrinkage estimation in partially linear models

In the context of a partially linear regression model, shrinkage semiparametric estimation is considered based on the Stein-rule. In this framework, the coefficient vector is partitioned into two sub-vectors: the first sub-vector gives the coefficients of interest, i.e., main effects (for example, treatment effects), and the second sub-vector is for variables that may or may not need to be controlled. When estimating the first sub-vector, the best estimate may be obtained using either the full model that includes both sub-vectors, or the reduced model which leaves out the second sub-vector. It is demonstrated that shrinkage estimators which combine two semiparametric estimators computed for the full model and the reduced model outperform the semiparametric estimator for the full model. Using the semiparametric estimate for the reduced model is best when the second sub-vector is the null vector, but this estimator suffers seriously from bias otherwise. The relative dominance picture of suggested estimators is investigated. In particular, suitability of estimating the nonparametric component based on the B-spline basis function is explored. Further, the performance of the proposed estimators is compared with an absolute penalty estimator through Monte Carlo simulation. Lasso and adaptive lasso were implemented for simultaneous model selection and parameter estimation. A real data example is given to compare the proposed estimators with lasso and adaptive lasso estimators.     

Estimation of probabilities using the logistic model in retrospective studies

Methods for estimating the parameters of the logistic regression model when the data are collected using a case-control (retrospective) scheme are compared. The regression coefficients are estimated by maximum likelihood methodology. This leaves the constant term parameter to be estimated. Four methods for estimating this parameter are proposed. The comparison of the four estimators is in two parts. First, they are compared for large samples. This is accomplished via the asymptotic distribution of the estimators. Second, the estimators are compared for small samples. This is conducted via stimulation using 11 logistic models. The estimation of the posterior probability of the response variable being a success (Px), as given by the logistic regression model, when the constant parameter is estimated by each of the four proposed methods is the main focus of this paper. A third concern is the comparison of the logistic discriminant procedures when each of the four methods of estimating the constant parameters is used. In addition, the linear discriminant function procedure is included. This comparison is executed only for small samples via simulation. It was found that when estimating Px, method 1 (which is essentially the MLE) minimizes the expected mean square error. The results were not as clear when the parameter of interest was the constant term itself. The results from the classification comparisons implied that when the logistic model contains mostly (or all) binary regression variables the logistic discriminant procedure using method 1 to estimate the constant term gives minimum expected error rate; otherwise the linear discriminant function gives minimum expected error rate. In the latter case the logistic discriminant procedure (method 1 estimator of the constant term) is approximately as good.     

Semiparametric analysis of randomized response data with missing covariates in logistic regression

In this article, two semiparametric approaches are developed for analyzing randomized response data with missing covariates in logistic regression model. One of the two proposed estimators is an extension of the validation likelihood estimator of Breslow and Cain [Breslow, N.E., and Cain, K.C. 1988. Logistic regression for two-stage case-control data. Biometrika. 75, 11-20]. The other is a joint conditional likelihood estimator based on both validation and non-validation data sets. We present a large sample theory for the proposed estimators. Simulation results show that the joint conditional likelihood estimator is more efficient than the validation likelihood estimator, weighted estimator, complete-case estimator and partial likelihood estimator. We also illustrate the methods using data from a cable TV study.     

Identification of process and measurement noise covariance for state and parameter estimation using extended Kalman filter

The performance of Bayesian state estimators, such as the extended Kalman filter (EKF), is dependent on the accurate characterisation of the uncertainties in the state dynamics and in the measurements. The parameters of the noise densities associated with these uncertainties are, however, often treated as ‘tuning parameters’ and adjusted in an ad hoc manner while carrying out state and parameter estimation. In this work, two approaches are developed for constructing the maximum likelihood estimates (MLE) of the state and measurement noise covariance matrices from operating input-output data when the states and/or parameters are estimated using the EKF. The unmeasured disturbances affecting the process are either modelled as unstructured noise affecting all the states or as structured noise entering the process predominantly through known, but unmeasured inputs. The first approach is based on direct optimisation of the ML objective function constructed by using the innovation sequence generated from the EKF. The second approach - the extended EM algorithm - is a derivative-free method, that uses the joint likelihood function of the complete data, i.e. states and measurements, to compute the next iterate of the decision variables for the optimisation problem. The efficacy of the proposed approaches is demonstrated on a benchmark continuous fermenter system. The simulation results reveal that both the proposed approaches generate fairly accurate estimates of the noise covariances. Experimental studies on a benchmark laboratory scale heater-mixer setup demonstrate a marked improvement in the predictions of the EKF that uses the covariance estimates obtained from the proposed approaches.     

基于协方差配置的采样估计

在工程应用中,状态估计的指标要求常常表现为误差协方差的形式.在充分考虑系 统内采样特性的基础上,提出了采样估计协方差的定义和一种新的采样估计方法,目的在于 设计离散估计器使采样估计协方差达到指定值,从而获得满意的稳定状态估计性能.将此采 样估计问题等价地转化为一个虚拟离散系统的估计器设计问题,给出了期望估计器的存在条 件及显式表示.数值例子说明了方法的有效性.