Robust mean-squared error estimation in the presence of model uncertainties

We consider the problem of estimating an unknown parameter vector x in a linear model that may be subject to uncertainties, where the vector x is known to satisfy a weighted norm constraint. We first assume that the model is known exactly and seek the linear estimator that minimizes the worst-case mean-squared error (MSE) across all possible values of x. We show that for an arbitrary choice of weighting, the optimal minimax MSE estimator can be formulated as a solution to a semidefinite programming problem (SDP), which can be solved very efficiently. We then develop a closed form expression for the minimax MSE estimator for a broad class of weighting matrices and show that it coincides with the shrunken estimator of Mayer and Willke, with a specific choice of shrinkage factor that explicitly takes the prior information into account. Next, we consider the case in which the model matrix is subject to uncertainties and seek the robust linear estimator that minimizes the worst-case MSE across all possible values of x and all possible values of the model matrix. As we show, the robust minimax MSE estimator can also be formulated as a solution to an SDP. Finally, we demonstrate through several examples that the minimax MSE estimator can significantly increase the performance over the conventional least-squares estimator, and when the model matrix is subject to uncertainties, the robust minimax MSE estimator can lead to a considerable improvement in performance over the minimax MSE estimator.     

A competitive minimax approach to robust estimation of random parameters

We consider the problem of estimating, in the presence of model uncertainties, a random vector x that is observed through a linear transformation H and corrupted by additive noise. We first assume that both the covariance matrix of x and the transformation H are not completely specified and develop the linear estimator that minimizes the worst-case mean-squared error (MSE) across all possible covariance matrices and transformations H in the region of uncertainty. Although the minimax approach has enjoyed widespread use in the design of robust methods, we show that its performance is often unsatisfactory. To improve the performance over the minimax MSE estimator, we develop a competitive minimax approach for the case where H is known but the covariance of x is subject to uncertainties and seek the linear estimator that minimizes the worst-case regret, namely, the worst-case difference between the MSE attainable using a linear estimator, ignorant of the signal covariance, and the optimal MSE attained using a linear estimator that knows the signal covariance. The linear minimax regret estimator is shown to be equal to a minimum MSE (MMSE) estimator corresponding to a certain choice of signal covariance that depends explicitly on the uncertainty region. We demonstrate, through examples, that the minimax regret approach can improve the performance over both the minimax MSE approach and a "plug in" approach, in which the estimator is chosen to be equal to the MMSE estimator with an estimated covariance matrix replacing the true unknown covariance. We then show that although the optimal minimax regret estimator in the case in which the signal and noise are jointly Gaussian is nonlinear, we often do not lose much by restricting attention to linear estimators.     

Minimax MSE-ratio estimation with signal covariance uncertainties

In continuation to an earlier work, we further consider the problem of robust estimation of a random vector (or signal), with an uncertain covariance matrix, that is observed through a known linear transformation and corrupted by additive noise with a known covariance matrix. While, in the earlier work, we developed and proposed a competitive minimax approach of minimizing the worst-case mean-squared error (MSE) difference regret criterion, here, we study, in the same spirit, the minimum worst-case MSE ratio regret criterion, namely, the worst-case ratio (rather than difference) between the MSE attainable using a linear estimator, ignorant of the exact signal covariance, and the minimum MSE (MMSE) attainable by optimum linear estimation with a known signal covariance. We present the optimal linear estimator, under this criterion, in two ways: The first is as a solution to a certain semidefinite programming (SDP) problem, and the second is as an expression that is of closed form up to a single parameter whose value can be found by a simple line search procedure. We then show that the linear minimax ratio regret estimator can also be interpreted as the MMSE estimator that minimizes the MSE for a certain choice of signal covariance that depends on the uncertainty region. We demonstrate that in applications, the proposed minimax MSE ratio regret approach may outperform the well-known minimax MSE approach, the minimax MSE difference regret approach, and the "plug-in" approach, where in the latter, one uses the MMSE estimator with an estimated covariance matrix replacing the true unknown covariance.     

Linear minimax regret estimation of deterministic parameters with bounded data uncertainties

We develop a new linear estimator for estimating an unknown parameter vector x in a linear model in the presence of bounded data uncertainties. The estimator is designed to minimize the worst-case regret over all bounded data vectors, namely, the worst-case difference between the mean-squared error (MSE) attainable using a linear estimator that does not know the true parameters x and the optimal MSE attained using a linear estimator that knows x. We demonstrate through several examples that the minimax regret estimator can significantly increase the performance over the conventional least-squares estimator, as well as several other least-squares alternatives.     

Universal Weighted MSE Improvement of the Least-Squares Estimator

Since the seminal work of Stein in the 1950s, there has been continuing research devoted to improving the total mean-squared error (MSE) of the least-squares (LS) estimator in the linear regression model. However, a drawback of these methods is that although they improve the total MSE, they do so at the expense of increasing the MSE of some of the individual signal components. Here we consider a framework for developing linear estimators that outperform the LS strategy over bounded norm signals, under all weighted MSE measures. This guarantees, for example, that both the total MSE and the MSE of each of the elements will be smaller than that resulting from the LS approach. We begin by deriving an easily verifiable condition on a linear method that ensures LS domination for every weighted MSE. We then suggest a minimax estimator that minimizes the worst-case MSE over all weighting matrices and bounded norm signals subject to the universal weighted MSE domination constraint.     

Minimax MSE estimation of deterministic parameters with noise covariance uncertainties

In this paper, a minimax mean-squared error (MSE) estimator is developed for estimating an unknown deterministic parameter vector in a linear model, subject to noise covariance uncertainties. The estimator is designed to minimize the worst-case MSE across all norm-bounded parameter vectors, and all noise covariance matrices, in a given region of uncertainty. The minimax estimator is shown to have the same form as the estimator that minimizes the worst-case MSE over all norm-bounded vectors for a least-favorable choice of the noise covariance matrix. An example demonstrating the performance advantage of the minimax MSE approach over the least-squares and weighted least-squares methods is presented.     

Comparing between estimation approaches: admissible and dominating linear estimators

We treat the problem of evaluating the performance of linear estimators for estimating a deterministic parameter vector x in a linear regression model, with the mean-squared error (MSE) as the performance measure. Since the MSE depends on the unknown vector x, a direct comparison between estimators is a difficult problem. Here, we consider a framework for examining the MSE of different linear estimation approaches based on the concepts of admissible and dominating estimators. We develop a general procedure for determining whether or not a linear estimator is MSE admissible, and for constructing an estimator strictly dominating a given inadmissible method so that its MSE is smaller for all x. In particular, we show that both problems can be addressed in a unified manner for arbitrary constraint sets on x by considering a certain convex optimization problem. We then demonstrate the details of our method for the case in which x is constrained to an ellipsoidal set and for unrestricted choices of x. As a by-product of our results, we derive a closed-form solution for the minimax MSE estimator on an ellipsoid, which is valid for arbitrary model parameters, as long as the signal-to-noise-ratio exceeds a certain threshold.     

Robust deconvolution of deterministic and random signals

We consider the problem of designing an estimation filter to recover a signal x[n] convolved with a linear time-invariant (LTI) filter h[n] and corrupted by additive noise. Our development treats the case in which the signal x[n] is deterministic and the case in which it is a stationary random process. Both formulations take advantage of some a priori knowledge on the class of underlying signals. In the deterministic setting, the signal is assumed to have bounded (weighted) energy; in the stochastic setting, the power spectra of the signal and noise are bounded at each frequency. The difficulty encountered in these estimation problems is that the mean-squared error (MSE) at the output of the estimation filter depends on the problem unknowns and therefore cannot be minimized. Beginning with the deterministic setting, we develop a minimax MSE estimation filter that minimizes the worst case point-wise MSE between the true signal x[n] and the estimated signal, over the class of bounded-norm inputs. We then establish that the MSE at the output of the minimax MSE filter is smaller than the MSE at the output of the conventional inverse filter, for all admissible signals. Next we treat the stochastic scenario, for which we propose a minimax regret estimation filter to deal with the power spectrum uncertainties. This filter is designed to minimize the worst case difference between the MSE in the presence of power spectrum uncertainties, and the MSE of the Wiener filter that knows the correct power spectra. The minimax regret filter takes the entire uncertainty interval into account, and as demonstrated through an example, can often lead to improved performance over traditional minimax MSE approaches for this problem     

Robust Competitive Estimation With Signal and Noise Covariance Uncertainties

Robust estimation of a random vector in a linear model in the presence of model uncertainties has been studied in several recent works. While previous methods considered the case in which the uncertainty is in the signal covariance, and possibly the model matrix, but the noise covariance is assumed to be completely specified, here we extend the results to the case where the noise statistics may also be subjected to uncertainties. We propose several different approaches to robust estimation, which differ in their assumptions on the given statistics. In the first method, we assume that the model matrix and both the signal and the noise covariance matrices are uncertain, and develop a minimax mean-squared error (MSE) estimator that minimizes the worst case MSE in the region of uncertainty. The second strategy assumes that the model matrix is given and tries to uniformly approach the performance of the linear minimum MSE estimator that knows the signal and noise covariances by minimizing a worst case regret measure. The regret is defined as the difference or ratio between the MSE attainable using a linear estimator, ignorant of the signal and noise covariances, and the minimum MSE possible when the statistics are known. As we show, earlier solutions follow directly from our more general results. However, the approach taken here in developing the robust estimators is considerably simpler than previous methods     

A Minimax Chebyshev Estimator for Bounded Error Estimation

We develop a nonlinear minimax estimator for the classical linear regression model assuming that the true parameter vector lies in an intersection of ellipsoids. We seek an estimate that minimizes the worst-case estimation error over the given parameter set. Since this problem is intractable, we approximate it using semidefinite relaxation, and refer to the resulting estimate as the relaxed Chebyshev center (RCC). We show that the RCC is unique and feasible, meaning it is consistent with the prior information. We then prove that the constrained least-squares (CLS) estimate for this problem can also be obtained as a relaxation of the Chebyshev center, that is looser than the RCC. Finally, we demonstrate through simulations that the RCC can significantly improve the estimation error over the CLS method.     

Blind Minimax Estimation

We consider the linear regression problem of estimating an unknown, deterministic parameter vector based on measurements corrupted by colored Gaussian noise. We present and analyze blind minimax estimators (BMEs), which consist of a bounded parameter set minimax estimator, whose parameter set is itself estimated from measurements. Thus, our approach does not require any prior assumption or knowledge, and the proposed estimator can be applied to any linear regression problem. We demonstrate analytically that the BMEs strictly dominate the least-squares (LS) estimator, i.e., they achieve lower mean-squared error (MSE) for any value of the parameter vector. Both Stein's estimator and its positive-part correction can be derived within the blind minimax framework. Furthermore, our approach can be readily extended to a wider class of estimation problems than Stein's estimator, which is defined only for white noise and nontransformed measurements. We show through simulations that the BMEs generally outperform previous extensions of Stein's technique.     

Minimum variance in biased estimation: bounds and asymptotically optimal estimators

We develop a uniform Cramer-Rao lower bound (UCRLB) on the total variance of any estimator of an unknown vector of parameters, with bias gradient matrix whose norm is bounded by a constant. We consider both the Frobenius norm and the spectral norm of the bias gradient matrix, leading to two corresponding lower bounds. We then develop optimal estimators that achieve these lower bounds. In the case in which the measurements are related to the unknown parameters through a linear Gaussian model, Tikhonov regularization is shown to achieve the UCRLB when the Frobenius norm is considered, and the shrunken estimator is shown to achieve the UCRLB when the spectral norm is considered. For more general models, the penalized maximum likelihood (PML) estimator with a suitable penalizing function is shown to asymptotically achieve the UCRLB. To establish the asymptotic optimality of the PML estimator, we first develop the asymptotic mean and variance of the PML estimator for any choice of penalizing function satisfying certain regularity constraints and then derive a general condition on the penalizing function under which the resulting PML estimator asymptotically achieves the UCRLB. This then implies that from all linear and nonlinear estimators with bias gradient whose norm is bounded by a constant, the proposed PML estimator asymptotically results in the smallest possible variance.     

A Competitive Mean-Squared Error Approach to Beamforming

We treat the problem of beamforming for signal estimation where the goal is to estimate a signal amplitude from a set of array observations. Conventional beamforming methods typically aim at maximizing the signal-to-interference-plus-noise ratio (SINR). However, this does not guarantee a small mean-squared error (MSE), so that on average the resulting signal estimate can be far from the true signal. Here, we consider strategies that attempt to minimize the MSE between the estimated and unknown signal waveforms. The methods we suggest all maximize the SINR but at the same time are designed to have good MSE performance. Since the MSE depends on the signal power, which is unknown, we develop competitive beamforming approaches that minimize a robust MSE measure. Two design strategies are proposed: minimax MSE and minimax regret. We demonstrate through numerical examples that the suggested minimax beamformers can outperform several existing standard and robust methods, over a wide range of signal-to-noise ratio (SNR) values. Finally, we apply our techniques to subband beamforming and illustrate their advantage in estimating a wideband signal.     

Minimax estimation of unknown deterministic signals in colorednoise

The estimation of a deterministic signal corrupted by random noise is considered. The strategy is to find a linear noncausal estimator which minimizes the maximum mean square error over an a priori set of signals. This signal set is specified in terms of frequency/energy constraints via the discrete Fourier transform. Exact filter expressions are given for the case of additive white noise. For the case of additive colored noise possessing a continuous power spectral density, a suboptimal filter is derived whose asymptotic performance is optimal. Asymptotic expressions for the minimax estimator error are developed for both cases. The minimax filter is applied to random data and is shown to solve asymptotically a certain worst-case Wiener filter problem     

State Estimation With Initial State Uncertainty

The problem of state estimation with initial state uncertainty is approached from a statistical decision theory point of view. The initial state is regarded as deterministic and unknown. It is only known that the initial state vector belongs to a specified parameter set. The (frequentist) risk is considered as the performance measure and the minimax approach is adopted. Minimax estimators are derived for some important cases of unbounded parameter sets. If the parameter set is bounded, a method of finding estimators whose maximum risk is arbitrarily close to that of a minimax estimator is provided. This method is illustrated with an example in which an estimator whose maximum risk is at most 3% larger than that of a minimax estimator is derived.     

Uniformly Improving the CramÉr-Rao Bound and Maximum-Likelihood Estimation

An important aspect of estimation theory is characterizing the best achievable performance in a given estimation problem, as well as determining estimators that achieve the optimal performance. The traditional CramÉr–Rao type bounds provide benchmarks on the variance of any estimator of a deterministic parameter vector under suitable regularity conditions, while requiring a-priori specification of a desired bias gradient. In applications, it is often not clear how to choose the required bias. A direct measure of the estimation error that takes both the variance and the bias into account is the mean squared error (MSE), which is the sum of the variance and the squared-norm of the bias. Here, we develop bounds on the MSE in estimating a deterministic parameter vector$ bf x_0$over all bias vectors that are linear in$ bf x_0$, which includes the traditional unbiased estimation as a special case. In some settings, it is possible to minimize the MSE over all linear bias vectors. More generally, direct minimization is not possible since the optimal solution depends on the unknown$ bf x_0$. Nonetheless, we show that in many cases, we can find bias vectors that result in an MSE bound that is smaller than the CramÉr–Rao lower bound (CRLB) for all values of$ bf x_0$. Furthermore, we explicitly construct estimators that achieve these bounds in cases where an efficient estimator exists, by performing a simple linear transformation on the standard maximum likelihood (ML) estimator. This leads to estimators that result in a smaller MSE than the ML approach for all possible values of$ bf x_0$.     

Estimation of Constrained Parameters With Guaranteed MSE Improvement

We address the problem of estimating an unknown parameter vector x in a linear model y=Cx+v subject to the a priori information that the true parameter vector x belongs to a known convex polytope X. The proposed estimator has the parametrized structure of the maximum a posteriori probability (MAP) estimator with prior Gaussian distribution, whose mean and covariance parameters are suitably designed via a linear matrix inequality approach so as to guarantee, for any xisinX, an improvement of the mean-squared error (MSE) matrix over the least-squares (LS) estimator. It is shown that this approach outperforms existing "superefficient" estimators for constrained parameters based on different parametrized structures and/or shapes of the parameter membership region X     

Linear Regression With Gaussian Model Uncertainty: Algorithms and Bounds

In this paper, we consider the problem of estimating an unknown deterministic parameter vector in a linear regression model with random Gaussian uncertainty in the mixing matrix. We prove that the maximum-likelihood (ML) estimator is a (de)regularized least squares estimator and develop three alternative approaches for finding the regularization parameter that maximizes the likelihood. We analyze the performance using the Cramer-Rao bound (CRB) on the mean squared error, and show that the degradation in performance due the uncertainty is not as severe as may be expected. Next, we address the problem again assuming that the variances of the noise and the elements in the model matrix are unknown and derive the associated CRB and ML estimator. We compare our methods to known results on linear regression in the error in variables (EIV) model. We discuss the similarity between these two competing approaches, and provide a thorough comparison that sheds light on their theoretical and practical differences.     

Nonlinear and Linear Broadcasting With QoS Requirements: Tractable Approaches for Bounded Channel Uncertainties

We consider the downlink of a cellular system in which the base station employs multiple transmit antennas, each receiver has a single antenna, and the users specify certain quality of service (QoS) requirements. We study the design of robust broadcasting schemes that minimize the transmission power necessary to guarantee that the QoS requirements are satisfied for all channels within bounded uncertainty regions around the transmitter's estimate of each user's channel. Each user's QoS requirement is formulated as a constraint on the mean square error (MSE) in its received signal, and we show that these MSE constraints imply constraints on the received signal-to-interference-plus-noise ratio. Using the MSE constraints, we present a unified approach to the design of linear and nonlinear transceivers with QoS requirements that must be satisfied in the presence of bounded channel uncertainty. The proposed designs overcome the limitations of existing approaches that provide conservative designs or are only applicable to the case of linear precoding. Furthermore, we provide computationally efficient design formulations for a rather general model of bounded channel uncertainty that subsumes many natural choices for the uncertainty region. We also consider the problem of the robust counterpart to precoding schemes that maximize the fidelity of the weakest user's signal subject to a power constraint. For this problem, we provide quasi-convex formulations, for both linear and nonlinear transceivers, that can be efficiently solved using a one-dimensional bisection search. Our numerical results demonstrate that in the presence of bounded uncertainty in the transmitter's knowledge of users' channels, the proposed designs provide guarantees for a larger range of QoS requirements than the existing approaches that are based on bounded channel uncertainty models and require less transmission power to provide these guarantees.     

Stochastic MV-PURE Estimator— Robust Reduced-Rank Estimator for Stochastic Linear Model

This paper proposes a novel linear estimator named stochastic MV-PURE estimator, developed for the stochastic linear model, and designed to provide improved performance over the linear minimum mean square error (MMSE) Wiener estimator in cases prevailing in practical, real-world settings, where at least some of the second-order statistics of the random vectors under consideration are only imperfectly known. The proposed estimator shares its main mathematical idea and terminology with the recently introduced minimum-variance pseudo-unbiased reduced-rank estimator (MV-PURE), developed for the linear regression model. The proposed stochastic MV-PURE estimator minimizes the mean square error (MSE) of its estimates subject to rank constraint and inducing minimium distortion to the target random vector. Therefore, the stochastic MV-PURE combines the techniques of the reduced rank Wiener filter (named in this paper RR-MMSE) and the distortionless-constrained estimator (named in this paper C-MMSE), in order to achieve greater robustness against noise or model errors than RR-MMSE and C-MMSE. Furthermore, to ensure that the stochastic MV-PURE estimator combines the reduced-rank and minimum-distortion approaches in the MSE-optimal way, we propose a rank selection criterion which minimizes the MSE of the estimates obtained by the stochastic MV-PURE. As a numerical example, we employ the stochastic MV-PURE, RR-MMSE, C-MMSE, and MMSE estimators as linear receivers in a MIMO wireless communication system. This example is chosen as a typical signal processing scenario, where the statistical information on the data, on which the estimates are built, is only imperfectly known. We verify that the stochastic MV-PURE achieves the lowest MSE and symbol error rate (SER) in such settings by employing the proposed rank selection criterion.