The extended least squares criterion: minimization algorithms andapplications

The least squares (LS) estimation criterion on one hand, and the total LS (TLS), constrained TLS (CTLS) and structured TLS (STLS) criteria on the other hand, can be viewed as opposite limiting cases of a more general criterion, which we term “extended LS” (XLS). The XLS criterion distinguishes measurement errors from modeling errors by properly weighting and balancing the two error sources. In the context of certain models (termed “pseudo-linear”), we derive two iterative algorithms for minimizing the XLS criterion: One is a straightforward “alternating coordinates” minimization, and the other is an extension of an existing CTLS algorithm. The algorithms exhibit different tradeoffs between convergence rate, computational load, and accuracy. The XLS criterion can be applied to popular estimation problems, such as identifying an autoregressive (AR) with exogenous noise (ARX) system from noisy input/output measurements or estimating the parameters of an AR process from noisy measurements. We demonstrate the convergence properties and performance of the algorithms with examples of the latter     

End-to-end color printer calibration by total least squaresregression

Neugebauer (1937) modeling plays an important role in obtaining end-to-end device characterization profiles for halftone color printer calibration. This paper proposes total least square (TLS) regression methods to estimate the parameters of various Neugebauer models. Compared to the traditional least squares (LS) based methods, the TLS approach is physically more appropriate for the printer modeling problem because it accounts for errors in the measured reflectance of both the primaries and the modeled samples. A TLS method based on print measurements from single-colorant step-wedges is first developed. The method is then extended to incorporate multicolorant print measurements using an iterative algorithm. The LS and TLS techniques are compared through tests performed on two color printers, one employing conventional rotated halftone screens and the other using a dot-on-dot halftone screen configuration. Our experiments indicate that the TLS methods yield a consistent and significant improvement over the LS-based techniques for model parameter estimation. The gains from the TLS method are particularly significant when the number of patches for which measured data is available is limited.     

Maximum likelihood estimation of signals in autoregressive noise

Time series modeling as the sum of a deterministic signal and an autoregressive (AR) process is studied. Maximum likelihood estimation of the signal amplitudes and AR parameters is seen to result in a nonlinear estimation problem. However, it is shown that for a given class of signals, the use of a parameter transformation can reduce the problem to a linear least squares one. For unknown signal parameters, in addition to the signal amplitudes, the maximization can be reduced to one over the additional signal parameters. The general class of signals for which such parameter transformations are applicable, thereby reducing estimator complexity drastically, is derived. This class includes sinusoids as well as polynomials and polynomial-times-exponential signals. The ideas are based on the theory of invariant subspaces for linear operators. The results form a powerful modeling tool in signal plus noise problems and therefore find application in a large variety of statistical signal processing problems. The authors briefly discuss some applications such as spectral analysis, broadband/transient detection using line array data, and fundamental frequency estimation for periodic signals     

QR-based TLS and mixed LS-TLS algorithms with applications to adaptive IIR filtering

The least squares (LS), total least squares (TLS), and mixed LS-TLS approaches are compared as to their properties and performance on several classical filtering problems. Mixed LS-TLS is introduced as a QR-decomposition-based algorithm for unbiased, equation error adaptive infinite impulse response (IIR) filtering. The algorithm is based on casting adaptive IIR filtering into a mixed LS-TLS framework. This formulation is shown to be equivalent to the minimization of the mean-square equation error subject to a unit norm constraint on the denominator parameter vector. An efficient implementation of the mixed LS-TLS solution is achieved through the use of back substitution and inverse iteration. Unbiasedness of the system parameter estimates is established for the mixed LS-TLS solution in the case of uncorrelated output noise, and the algorithm is shown to converge to this solution. LS, TLS, and mixed LS-TLS performance is then compared for the problems of echo cancellation, noise reduction, and frequency equalization.     

Modified LMS algorithm for unbiased impulse response estimation innonstationary noise

In the presence of input interference, the Wiener solution for impulse response estimation is biased. It is proved that bias removal can be achieved by proper scaling of the optimal filter coefficients and a modified least mean squares (LMS) algorithm is then developed for accurate system identification in noise. Simulation results show that the proposed method outperforms two total least squares (TLS) based adaptive algorithms under nonstationary interference conditions     

Deterministic regression methods for unbiased estimation of time-varying autoregressive parameters from noisy observations

A great deal of interest has been paid to autoregressive parameter estimation in the noise-free case or when the observation data are disturbed by random noise. Tracking time-varying autoregressive (TVAR) parameters has been also discussed, but few papers deal with this issue when there is an additive zero-mean white Gaussian measurement noise. In this paper, one considers deterministic regression methods (or evolutive methods) where the TVAR parameters are assumed to be weighted combinations of basis functions. However, the additive white measurement noise leads to a weight-estimation bias when standard least squares methods are used. Therefore, we propose two alternative blind off-line methods that allow both the variance of the additive noise and the weights to be estimated. The first one is based on the errors-in-variable issue whereas the second consists in viewing the estimation issue as a generalized eigenvalue problem. A comparative study with other existing methods confirms the effectiveness of the proposed methods.     

Inverse filtering based method for estimation of noisy autoregressive signals

In this paper we present a new method for estimating the parameters of an autoregressive (AR) signal from observations corrupted with white noise. The least-squares (LS) estimate of the AR parameters is biased when the observation noise is added to the AR signal. This bias is related to observation noise variance. The proposed method uses inverse filtering technique and Yule-Walker equations for estimating observation noise variance to yield unbiased LS estimate of the AR parameters. The performance of the proposed unbiased algorithm is illustrated by simulation results and they show that the performance of the proposed method is better than the other estimation methods.     

Fast identification of autoregressive signals from noisy observations

The purpose of this brief is to derive, from the previously developed least-squares (LS) based method, a faster convergent approach to identification of noisy autoregressive (AR) stochastic signals. The feature of the new algorithm is that in its bias correction procedure, it makes use of more autocovariance samples to estimate the variance of the additive corrupting noise which determines the noise-induced bias in the LS estimates of the AR parameters. Since more accurate estimates of this corrupting noise variance can be attained at earlier stages of the iterative process, the proposed algorithm can achieve a faster rate of convergence. Simulation results are included that illustrate the good performances of the proposed algorithm.     

Consistent normalized least mean square filtering with noisy data matrix

When the ordinary least squares method is applied to the parameter estimation problem with noisy data matrix, it is well-known that the estimates turn out to be biased. While this bias term can be somewhat reduced by the use of models of higher order, or by requiring a high signal-to-noise ratio (SNR), it can never be completely removed. Consistent estimates can be obtained by means of the instrumental variable method (IVM),or the total/data least squares method (TLS/DLS). In the adaptive setting for the such problem, a variety of least-mean-squares (LMS)-type algorithms have been researched rather than their recursive versions of IVM or TLS/DLS that cost considerable computations. Motivated by these observations, we propose a consistent LMS-type algorithm for the data least square estimation problem. This novel approach is based on the geometry of the mean squared error (MSE) function, rendering the step-size normalization and the heuristic filtered estimation of the noise variance, respectively, for fast convergence and robustness to stochastic noise. Monte Carlo simulations of a zero-forcing adaptive finite-impulse-response (FIR) channel equalizer demonstrate the efficacy of our algorithm.     

Parameter estimation in chaotic noise

The problem of parameter estimation in chaotic noise is considered in this paper. Since a chaotic signal is inherently deterministic, a new complexity measure called the phase space volume (PSV) is introduced for estimation instead of using the conventional probabilistic measures. We show that the unknown parameters of a signal embedded in chaotic noise ran be obtained by minimizing the PSV (MPSV) of the output of an inverse filter of the received signal in a reconstructed phase space. Monte Carlo simulations are carried out to analyze the efficiency of the MPSV method for parameter estimation in chaotic noise. To illustrate the usefulness of the MPSV technique in solving real-life problems, the problem of sinusoidal frequency estimation in real radar clutter (unwanted radar backscatters) is considered. Modeling radar clutter as a chaotic process, we apply the MPSV technique to estimate the sinusoidal frequencies by estimating the coefficients of an autoregressive (AR) spectrum. The results show that the frequency estimates generated by the MPSV method are more accurate than those obtained by the standard least square (LS) technique     

基于扩展卡尔曼滤波的联合迭代检测译码信道估计方法

针对高速移动场景下信道快衰落、非平稳等特性导致下行链路信道估计性能受限的问题,提出了一种适用于高速移动环境下行链路的信道估计方法.采用自回归过程对信道建模,构造自反馈的扩展卡尔曼滤波器（Extended Kalman Filter,EKF）追踪信道响应及其时域相关系数.为了解决EKF自反馈结构引起的误差传播问题,采用了迭代检测译码的接收机结构,以利用信道编码的冗余提升EKF的信道估计精度.仿真分析表明,在高速移动环境下所提方法相较于最小二乘估计和线性最小均方误差估计等传统方法提升了信道估计的均方误差和系统的误码率性能,可应用于高速列车无线通信设备的接收机基带信号处理系统.     

Identification of image and blur parameters in frequency domainusing the EM algorithm

We extend a method presented previously, which considers the problem of the semicausal autoregressive (AR) parameter identification for images degraded by observation noise only. We propose a new approach to identify both the causal and semicausal AR parameters and blur parameters without a priori knowledge of the observation noise power and the PSF of the degradation. We decompose the image into 1-D independent complex scalar subsystems resulting from the vector state-space model by using the unitary discrete Fourier transform (DFT). Then, by applying the expectation-maximization (EM) algorithm to each subsystem, we identify the AR model and blur parameters of the transformed image. The AR parameters of the original image are then identified by using the least squares (LS) method. The restored image is obtained as a byproduct of the EM algorithm.     

The constrained total least squares technique and its applicationsto harmonic superresolution

The constrained total least squares (CTLS) method is a natural extension of TLS to the case when the noise components of the coefficients are algebraically related. The CTLS technique is developed, and some of its applications to superresolution harmonic analysis are presented. The CTLS problem is reduced to an unconstrained minimization problem over a small set of variables. A perturbation analysis of the CTLS solution is derived, and from it the root mean-square error (RMSE) of the CTLS solution, which is valid for small noise levels, is obtained in closed form. The complex version of the Newton method is derived and applied to determine the CTLS solution. It is also shown that the CTLS problem is equivalent to a constrained parameter maximum-likelihood problem. The CTLS technique is applied to estimate the frequencies of sinusoids in white noise and the angle of arrival of narrowband wavefronts at a linear uniform array. In both cases the CTLS method shows superior or similar accuracy to other advanced techniques     

Parameter identification and position control for helical hydraulic rotary actuators based on particle swarm optimization

It is difficult to obtain an accurate mathematical model in electro-hydraulic servo control system, due to the nonlinear factors such as dead zone, saturation, flow coefficient, and friction. Hence, a parameter identification algorithm, combining recursive least squares (RLS) with modified nonlinear particle swarm optimization (NPSO) algorithm, is proposed. On this basis, another improved NPSO algorithm is also put forward, aiming at searching for the optimal proportional–integral (PI) controller gain of the nonlinear hydraulic system while giving comprehensive consideration to the system performance indexes. The system identification experiments and position tracking control are conducted, respectively. As indicated by the comparison with the least squares (LS), RLS, PSO, and RLS–LPSO results, the proposed method shows higher identification and control accuracy.     

Recursive blind identification of non-Gaussian time-varying ARmodel and application to blind equalisation of time-varying channel

A novel method for the blind identification of a non-Gaussian time-varying autoregressive model is presented. By approximating the non-Gaussian probability density function of the model driving noise sequence with a Gaussian-mixture density, a pseudo maximum-likelihood estimation algorithm is proposed for model parameter estimation. The real model identification is then converted to a recursive least squares estimation of the model time-varying parameters and an inference of the Gaussian-mixture parameters, so that the entire identification algorithm can be recursively performed. As an important application, the proposed algorithm is applied to the problem of blind equalisation of a time-varying AR communication channel online. Simulation results show that the new blind equalisation algorithm can achieve accurate channel estimation and input symbol recovery     

基于总体最小二乘准则的OFDM系统时变信道估计

信道估计的准确程度直接影响正交频分复用系统的性能。为了提高时变信道估计算法的精度,基于总体最小二乘准则( TLS)提出了一种时变信道的估计方法。该方法用线性模型对时变信道进行建模,不仅考虑了信道噪声,同时也兼顾了模型误差。该方法能较好地跟踪信道的变化,显著消除模型误差。仿真结果表明所提算法的均方误差介于最小二乘算法与最小均方误差算法之间,在不同归一化多普勒频移下,该算法具有较好的稳健性。     

Recursive parameter estimation for noisy autoregressive signals (Corresp.)

The problem of recursively estimating the unknown parameters of a scalar autoregressive (AR) signal observed in additive white noise, including signal power and noise variance, is considered. A state-space model in a canonical but noninnovations form is used to represent the noisy AR signal. An algorithm based on a system identification/parameter estimation technique known as the recursive prediction error method is presented for recursive parameter estimation. Two simulation examples illustrate the effectiveness of the proposed algorithm.     

A unified approach to three eigendecomposition methods forfrequency estimation

The authors present a unified approach to three eigendecomposition-based methods for frequency estimation in the presence of noise. These are the Tufts-Kumaresan (TK) method, the minimum-norm (MN) method, and the total least squares (TLS) method. It is shown that: (1) the MN method is a modified version of the TK method; (2) the TLS method is a generalization of the MN method; (3) the TLS solution vector can be expressed in matrix form, and an alternative way of computing it is presented; (4) the MN and the TLS methods exhibit some improvement over the TK method     

Total least squares phased averaging and 3-D ESPRIT for jointazimuth-elevation-carrier estimation

The method of total least squares (TLS) phased averaging for high-performance subspace fitting in the three-dimensional (3-D) case of spectral estimation with 3-D ESPRIT is introduced and applied to the joint azimuth elevation-carrier estimation problem with two-dimensional (2-D) uniform rectangular arrays. The method is highly efficient computationally and is suitable for large arrays. Detailed computer experiments and comparisons are provided. For a 16×16 array of sensors and heavy noise, TLS phased-averaging 3-D ESPRIT exceeds the 3-D TLS unitary ESPRIT estimator by 300% in RMSE performance     

Formulation and solution of structured total least norm problemsfor parameter estimation

The total least squares (TLS) method is a generalization of the least squares (LS) method for solving overdetermined sets of linear equations Ax≈b. The TLS method minimizes ∥[E|-r]∥_F, where r=b-(A+E)x, so that (b-r)∈Range (A+E), given A∈C^m×n, with m⩾n and b∈C^m×1. The most common TLS algorithm is based on the singular value decomposition (SVD) of [A/b]. However, the SVD-based methods may not be appropriate when the matrix A has a special structure since they do not preserve the structure. Previously, a new problem formulation known as structured total least norm (STLN), and the algorithm for computing the STLN solution, have been developed. The STLN method preserves the special structure of A or [A/b] and can minimize the error in the discrete L_p norm, where p=1, 2 or ∞. In this paper, the STLN problem formulation is generalized for computing the solution of STLN problems with multiple right-hand sides AX≈B. It is shown that these problems can be converted to ordinary STLN problems with one right-hand side. In addition, the method is shown to converge to the optimal solution in certain model reduction problems. Furthermore, the application of the STLN method to various parameter estimation problems is studied in which the computed correction matrix applied to A or [A/B] keeps the same Toeplitz structure as the data matrix A of [A/B], respectively. In particular, the L₂ norm STLN method is compared with the LS and TLS methods in deconvolution, transfer function modeling, and linear prediction problems