Improved Stein-type shrinkage estimators for the high-dimensional multivariate normal covariance matrix

Many applications require an estimate for the covariance matrix that is non-singular and well-conditioned. As the dimensionality increases, the sample covariance matrix becomes ill-conditioned or even singular. A common approach to estimating the covariance matrix when the dimensionality is large is that of Stein-type shrinkage estimation. A convex combination of the sample covariance matrix and a well-conditioned target matrix is used to estimate the covariance matrix. Recent work in the literature has shown that an optimal combination exists under mean-squared loss, however it must be estimated from the data. In this paper, we introduce a new set of estimators for the optimal convex combination for three commonly used target matrices. A simulation study shows an improvement over those in the literature in cases of extreme high-dimensionality of the data. A data analysis shows the estimators are effective in a discriminant and classification analysis.     

The use of shrinkage estimators in linear discriminant analysis

Probably the most common single discriminant algorithm in use today is the linear algorithm. Unfortunately, this algorithm has been shown to frequently behave poorly in high dimensions relative to other algorithms, even on suitable Gaussian data. This is because the algorithm uses sample estimates of the means and covariance matrix which are of poor quality in high dimensions. It seems reasonable that if these unbiased estimates were replaced by estimates which are more stable in high dimensions, then the resultant modified linear algorithm should be an improvement. This paper studies using a shrinkage estimate for the covariance matrix in the linear algorithm. We chose the linear algorithm, not because we particularly advocate its use, but because its simple structure allows one to more easily ascertain the effects of the use of shrinkage estimates. A simulation study assuming two underlying Gaussian populations with common covariance matrix found the shrinkage algorithm to significantly outperform the standard linear algorithm in most cases. Several different means, covariance matrices, and shrinkage rules were studied. A nonparametric algorithm, which previously had been shown to usually outperform the linear algorithm in high dimensions, was included in the simulation study for comparison.     

Video Denoising via Empirical Bayesian Estimation of Space-Time Patches

In this paper we present a new patch-based empirical Bayesian video denoising algorithm. The method builds a Bayesian model for each group of similar space-time patches. These patches are not motion-compensated, and therefore avoid the risk of inaccuracies caused by motion estimation errors. The high dimensionality of spatiotemporal patches together with a limited number of available samples poses challenges when estimating the statistics needed for an empirical Bayesian method. We therefore assume that groups of similar patches have a low intrinsic dimensionality, leading to a spiked covariance model. Based on theoretical results about the estimation of spiked covariance matrices, we propose estimators of the eigenvalues of the a priori covariance in high-dimensional spaces as simple corrections of the eigenvalues of the sample covariance matrix. We demonstrate empirically that these estimators lead to better empirical Wiener filters. A comparison on classic benchmark videos demonstrates improved visual quality and an increased PSNR with respect to state-of-the-art video denoising methods.     

A New Approach To Optimal and Self‐Tuning Filtering,Smoothing and Prediction For Discrete‐Time Systems

A new approach to optimal and self‐tuning state estimation of linear discrete time‐invariant systems is presented, using projection theory and innovation analysis method in time domain. The optimal estimators are calculated by means of spectral factorization. The filter, predictor, and smoother are given in a unified form. Comparisons are made to the previously known techniques such as the Kalman filtering and the polynomial method initiated by Kucera. When the noise covariance matrices are not available, self‐tuning estimators are obtained through the identification of an ARMA innovation model. The self‐tuning estimator asymptotically converges to the optimal estimator.     

Empirical Bayes estimators of the random intercept in multilevel analysis: Performance of the classical, Morris and Rao version

For a linear multilevel model with 2 levels, with equal numbers of level-1 units per level-2 unit and a random intercept only, different empirical Bayes estimators of the random intercept are examined. Studied are the classical empirical Bayes estimator, the Morris version of the empirical Bayes estimator and Rao's estimator. It is unclear which of these estimators performs best in terms of Bayes risk. Of these three, the Rao estimator is optimal in case the covariance matrix of random coefficients may be negative definite. However, in the multilevel model this matrix is restricted to be positive semi-definite. The Morris version, replaces the weights of the empirical Bayes estimator by unbiased estimates. This correction, however, is based on known level-1 variances, which in many empirical settings are unknown. A fourth estimator is proposed, a variant of Rao's estimator which restricts the estimated covariance matrix of random coefficients to be positive semi-definite. Since there are no closed-form expressions for estimators involved in the empirical Bayes estimators (except for the Rao estimator), Monte Carlo simulations are done to evaluate the performance of these different empirical Bayes estimators. Only for small sample sizes there are clear differences between these estimators. As a consequence, for larger sample sizes the formula for the Bayes risk of the Rao estimator can be used to calculate the Bayes risk for the other estimators proposed.     

LMI-based minimax estimation and filtering under unknown covariances

In this paper, we consider a minimax approach to the estimation and filtering problems in the stochastic framework, where covariances of the random factors are completely unknown. The term ‘random factors’ refers either to unknown parameters and measurement noise in the estimation problem or to disturbance process and the initial state of a linear discrete-time dynamic system in the filtering problem. We introduce a notion of the attenuation level of random factors as a performance measure for both a linear unbiased estimate and a filter. This is the worst-case variance of the estimation error normalised by the sum of variances of all random factors over all nonzero covariance matrices. It is shown that this performance measure is equal to the spectral norm of the ‘transfer matrix’ and therefore the minimax estimate and filter can be computed in terms of linear matrix inequalities (LMIs). Moreover, the explicit formulae for both the minimax estimate and the minimal value of the attenuation level are presented in the estimation problem. It turns out that the above attenuation level of random factors coincides with the attenuation level of deterministic factors that is the worst-case normalised squared Euclidian norm of the estimation error over all nonzero sample values of random factors. In addition, we demonstrate that the LMI technique can be applied to derive the optimal robust estimator and filter, when there is a priori information about convex polyhedral sets which unknown covariance matrices of random factors belong to. Two illustrative examples show advantages of the minimax approach proposed.     

State estimation for stochastic time‐varying multisensor systems with multiplicative noises: Centralized and decentralized data fusion

In this paper, the state estimation problems, including filtering and one‐step prediction, are solved for uncertain stochastic time‐varying multisensor systems by using centralized and decentralized data fusion methods. Uncertainties are considered in all parts of the state space model as multiplicative noises. For the first time, both centralized and decentralized estimators are designed based on the regularized least‐squares method. To design the proposed centralized fusion estimator, observation equations are first rewritten as a stacked observation. Then, an optimal estimator is obtained from a regularized least‐squares problem. In addition, for decentralized data fusion, first, optimal local estimators are designed, and then fusion rule is achieved by solving a least‐squares problem. Two recursive equations are also obtained to compute the unknown covariance matrices of the filtering and prediction errors. Finally, a three‐sensor target‐tracking system is employed to demonstrate the effectiveness and performance of the proposed estimation approaches.     

Stable estimation of a covariance matrix guided by nuclear norm penalties

Estimation of a covariance matrix or its inverse plays a central role in many statistical methods. For these methods to work reliably, estimated matrices must not only be invertible but also well-conditioned. The current paper introduces a novel prior to ensure a well-conditioned maximum a posteriori (MAP) covariance estimate. The prior shrinks the sample covariance estimator towards a stable target and leads to a MAP estimator that is consistent and asymptotically efficient. Thus, the MAP estimator gracefully transitions towards the sample covariance matrix as the number of samples grows relative to the number of covariates. The utility of the MAP estimator is demonstrated in two standard applications–discriminant analysis and EM clustering–in challenging sampling regimes.     

Nonparametric discriminant analysis

A nonparametric method of discriminant analysis is proposed. It is based on nonparametric extensions of commonly used scatter matrices. Two advantages result from the use of the proposed nonparametric scatter matrices. First, they are generally of full rank. This provides the ability to specify the number of extracted features desired. This is in contrast to parametric discriminant analysis, which for an L class problem typically can determine at most L 1 features. Second, the nonparametric nature of the scatter matrices allows the procedure to work well even for non-Gaussian data sets. Using the same basic framework, a procedure is proposed to test the structural similarity of two distributions. The procedure works in high-dimensional space. It specifies a linear decomposition of the original data space in which a relative indication of dissimilarity along each new basis vector is provided. The nonparametric scatter matrices are also used to derive a clustering procedure, which is recognized as a k-nearest neighbor version of the nonparametric valley seeking algorithm. The form which results provides a unified view of the parametric nearest mean reclassification algorithm and the nonparametric valley seeking algorithm.     

利用粒子群优化估计高光谱数据协方差矩阵

与传统的多光谱遥感相比,高光谱遥感具有更高的光谱分辨率,能更好地进行地物分类识别。但是,当训练样本数与数据维数相当,或小于后者时,会导致协方差矩阵近似奇异或奇异,使得经典最大似然分类失效,需要对协方差矩阵进行修正。典型的协方差阵估计方法往往只选取总体协方差、类别协方差及其相应变形中的两种形式进行组合,未考虑多种形式共同对协方差阵估计的影响。提出将PSO算法应用到协方差阵估计中,考虑所有形式的共同作用,对组合参数进行优化。最后,通过高光谱数据的分类实验证明了方法的可行性和有效性。     

Covariance matrix estimation for left-censored data

Multivariate methods often rely on a sample covariance matrix. The conventional estimators of a covariance matrix require complete data vectors on all subjects—an assumption that can frequently not be met. For example, in many fields of life sciences that are utilizing modern measuring technology, such as mass spectrometry, left-censored values caused by denoising the data are a commonplace phenomena. Left-censored values are low-level concentrations that are considered too imprecise to be reported as a single number but known to exist somewhere between zero and the laboratory’s lower limit of detection. Maximum likelihood-based covariance matrix estimators that allow the presence of the left-censored values without substituting them with a constant or ignoring them completely are considered. The presented estimators efficiently use all the information available and thus, based on simulation studies, produce the least biased estimates compared to often used competing estimators. As the genuine maximum likelihood estimate can be solved fast only in low dimensions, it is suggested to estimate the covariance matrix element-wise and then adjust the resulting covariance matrix to achieve positive semi-definiteness. It is shown that the new approach succeeds in decreasing the computation times substantially and still produces accurate estimates. Finally, as an example, a left-censored data set of toxic chemicals is explored.     

Determination of the optimal number of features for quadratic discriminant analysis via the normal approximation to the discriminant distribution

Given the joint feature-label distribution, increasing the number of features always results in decreased classification error; however, this is not the case when a classifier is designed via a classification rule from sample data. Typically, for fixed sample size, the error of a designed classifier decreases and then increases as the number of features grows. The problem is especially acute when sample sizes are very small and the potential number of features is very large. To obtain a general understanding of the kinds of feature-set sizes that provide good performance for a particular classification rule, performance must be evaluated based on accurate error estimation, and hence a model-based setting for optimizing the number of features is needed. This paper treats quadratic discriminant analysis (QDA) in the case of unequal covariance matrices. For two normal class-conditional distributions, the QDA classifier is determined according to a discriminant. The standard plug-in rule estimates the discriminant from a feature-label sample to obtain an estimate of the discriminant by replacing the means and covariance matrices by their respective sample means and sample covariance matrices. The unbiasedness of these estimators assures good estimation for large samples, but not for small samples.Our goal is to find an essentially analytic method to produce an error curve as a function of the number of features so that the curve can be minimized to determine an optimal number of features. We use a normal approximation to the distribution of the estimated discriminant. Since the mean and variance of the estimated discriminant will be exact, these provide insight into how the covariance matrices affect the optimal number of features. We derive the mean and variance of the estimated discriminant and compare feature-size optimization using the normal approximation to the estimated discriminant with optimization obtained by simulating the true distribution of the estimated discriminant. Optimization via the normal approximation to the estimated discriminant provides huge computational savings in comparison to optimization via simulation of the true distribution. Feature-size optimization via the normal approximation is very accurate when the covariance matrices differ modestly. The optimal number of features based on the normal approximation will exceed the actual optimal number when there is large disagreement between the covariance matrices; however, this difference is not important because the true misclassification error using the number of features obtained from the normal approximation and the number obtained from the true distribution differ only slightly, even for significantly different covariance matrices.     

基于分箱核密度估计的非参数多模态背景模型

提出了一种用于检测运动目标的非参数多模态背景模型。该模型采用分箱核密度估计算法从训练图像序列中得到背景的密度函数。分箱核密度估计算法利用基于网格数据重心的分箱规则,很好地提取了训练图像序列的关键信息,避免了采用全样本数据点的重复计算, 大大提高了运动目标检测算法的实时性。通过与全样本算法进行对比,发现该背景模型在运动目标检测中的有效性,可用于户外的实时交通监控系统。     

Optimal multilinear estimation of a random vector under constraints of causality and limited memory

A new technique is provided for random vector estimation from noisy data under the constraints that the estimator is causal and dependent on at most a finite number p of observations. Nonlinear estimators defined by multilinear operators of degree r are employed, the choice of r allowing a trade-off between the accuracy of the optimal filter and the complexity of the calculations. The techniques utilise an exact correspondence of the nonlinear problem to a corresponding linear one. This is then solved by a new procedure, the least squares singular pivot algorithm, whereby the linear problem can be repeated reduced to smaller structurally similar problems. Invertibility of the relevant covariance matrices is not assumed. Numerical experiments with real data are used to illustrate the efficacy of the new algorithm.     

自校正解耦信息融合Wiener状态估值器

对含未知噪声方差阵的多传感器系统,用现代时间序列分析方法.基于滑动平均(MA)新息模型的在线辨识和求解相关函数矩阵方程组,可得到估计噪声方差阵估值器,进而在按分量标量加权线性最小方差最优信息融合则下,提出了自校正解耦信息融合Wiener状态估值器.它的精度比每个局部自校正Wiener状态估值器精度高.它实现了状态分量的解耦局部Wiener估值器和解耦融合Wiener估值器.证明了它的收敛性,即若MA新息模型参数估计是一致的,则它将收敛于噪声统计已知时的最优解耦信息融合Wiener状态估值器,因而它具有渐近最优性.一个带3传感器的目标跟踪系统的仿真例子说明了其有效性.     

Dimension reduction by a novel unified scheme using divergence analysis and genetic search

In this study, a unified scheme using divergence analysis and genetic search is proposed to determine significant components of feature vectors in high-dimensional spaces, without having to deal with singular matrix problems.In the literature it is observed that three main problems exist in the feature selection process performed in a high-dimensional space. These problems are high computational load, local minima, and singular matrices. In this study, feature selection is realized by increasing the dimension one by one, rather than reducing the dimension. In this sense, the recursive covariance matrices are formulated to decrease the computational load. The use of genetic algorithms is proposed to avoid local optima and singular matrix problems in high-dimensional feature spaces. Candidate strings in the genetic pool represent the new features formed by increasing the dimension. The genetic algorithms investigate the combination of features which give the highest divergence value.In this study, two methods are proposed for the selection of features. In the first method, features in a high-dimensional space are determined by using divergence analysis and genetic search (DAGS) together. If the dimension is not high, the second method is offered which uses only recursive divergence analysis (RDA) without any genetic search. In Section 3 two experiments are presented: Feature determination in a two-dimensional phantom feature space, and feature determination for ECG beat classification in a real data space.     

On the sampling distribution of resubstitution and leave-one-out error estimators for linear classifiers

Error estimation is a problem of high current interest in many areas of application. This paper concerns the classical problem of determining the performance of error estimators in small-sample settings under a Gaussianity parametric assumption. We provide here for the first time the exact sampling distribution of the resubstitution and leave-one-out error estimators for linear discriminant analysis (LDA) in the univariate case, which is valid for any sample size and combination of parameters (including unequal variances and sample sizes for each class). In the multivariate case, we provide a quasi-binomial approximation to the distribution of both the resubstitution and leave-one-out error estimators for LDA, under a common but otherwise arbitrary class covariance matrix, which is assumed to be known in the design of the LDA discriminant. We provide numerical examples, using both synthetic and real data, that indicate that these approximations are accurate, provided that LDA classification error is not too large.     

Optimal convex error estimators for classification

A cross-validation error estimator is obtained by repeatedly leaving out some data points, deriving classifiers on the remaining points, computing errors for these classifiers on the left-out points, and then averaging these errors. The 0.632 bootstrap estimator is obtained by averaging the errors of classifiers designed from points drawn with replacement and then taking a convex combination of this “zero bootstrap” error with the resubstitution error for the designed classifier. This gives a convex combination of the low-biased resubstitution and the high-biased zero bootstrap. Another convex error estimator suggested in the literature is the unweighted average of resubstitution and cross-validation. This paper treats the following question: Given a feature-label distribution and classification rule, what is the optimal convex combination of two error estimators, i.e. what are the optimal weights for the convex combination. This problem is considered by finding the weights to minimize the MSE of a convex estimator. It also considers optimality under the constraint that the resulting estimator be unbiased. Owing to the large amount of results coming from the various feature-label models and error estimators, a portion of the results are presented herein and the main body of results appears on a companion website. In the tabulated results, each table treats the classification rules considered for the model, various Bayes errors, and various sample sizes. Each table includes the optimal weights, mean errors and standard deviations for the relevant error measures, and the MSE and MAE for the optimal convex estimator. Many observations can be made by considering the full set of experiments. Some general trends are outlined in the paper. The general conclusion is that optimizing the weights of a convex estimator can provide substantial improvement, depending on the classification rule, data model, sample size and component estimators. Optimal convex bootstrap estimators are applied to feature-set ranking to illustrate their potential advantage over non-optimized convex estimators.     

Joint estimation of mean-covariance model for longitudinal data with basis function approximations

When the selected parametric model for the covariance structure is far from the true one, the corresponding covariance estimator could have considerable bias. To balance the variability and bias of the covariance estimator, we employ a nonparametric method. In addition, as different mean structures may lead to different estimators of the covariance matrix, we choose a semiparametric model for the mean so as to provide a stable estimate of the covariance matrix. Based on the modified Cholesky decomposition of the covariance matrix, we construct the joint mean-covariance model by modeling the smooth functions using the spline method and estimate the associated parameters using the maximum likelihood approach. A simulation study and a real data analysis are conducted to illustrate the proposed approach and demonstrate the flexibility of the suggested model.     

Adaptive divided difference filtering for simultaneous state and parameter estimation

A novel adaptive version of the divided difference filter (DDF) applicable to non-linear systems with a linear output equation is presented in this work. In order to make the filter robust to modeling errors, upper bounds on the state covariance matrix are derived. The parameters of this upper bound are then estimated using a combination of offline tuning and online optimization with a linear matrix inequality (LMI) constraint, which ensures that the predicted output error covariance is larger than the observed output error covariance. The resulting sub-optimal, high-gain filter is applied to the problem of joint state and parameter estimation. Simulation results demonstrate the superior performance of the proposed filter as compared to the standard DDF.