A new interpretation on the MMSE as a robust MEE criterion

The minimum mean-square error (MMSE) and minimum error entropy (MEE) are two important criteria in the estimation related problems. The MMSE can be viewed as a robust MEE criterion in the minimax sense, as its minimization is equivalent to minimizing an upper bound (the maximum value) of the error entropy. This note gives a new and more meaningful interpretation on the robustness of MMSE for problems in which there exists uncertainty in the probability model. It is shown that the MMSE estimator imposes an upper bound on error entropy for the true model. The upper bound consists of two terms. The first term quantifies the “MMSE performance” under nominal conditions, and the second term measures the “distance” between the true and nominal models. This robustness property is parallel to that of the risk-sensitive estimation. Illustration examples are included to confirm the robustness of MMSE.     

Chrestenson最大熵谱估计的计算

本文研究p-平稳随机序列在Chrestenson变换意义下的功率谱密度及其最大熵谱估计的计算。得到了功率谱密度函数与熵率的关系式及最大熵谱估计的正规方程。在采样数据个数为p~m的情况下,最大熵谱估计可直接由已知的有限自相关数据表示。这些结果与Fourier意义下的最大熵谱估计有很大的不同。     

On the Estimation of Differential Entropy From Data Located on Embedded Manifolds

Estimation of the differential entropy from observations of a random variable is of great importance for a wide range of signal processing applications such as source coding, pattern recognition, hypothesis testing, and blind source separation. In this paper, we present a method for estimation of the Shannon differential entropy that accounts for embedded manifolds. The method is based on high-rate quantization theory and forms an extension of the classical nearest-neighbor entropy estimator. The estimator is consistent in the mean square sense and an upper bound on the rate of convergence of the estimator is given. Because of the close connection between compression and Shannon entropy, the proposed method has an advantage over methods estimating the Renyi entropy. Through experiments on uniformly distributed data on known manifolds and real-world speech data we show the accuracy and usefulness of our proposed method.     

Entropy minimization for supervised digital communications channelequalization

This paper investigates the application of error-entropy minimization algorithms to digital communications channel equalization. The pdf of the error between the training sequence and the output of the equalizer is estimated using the Parzen windowing method with a Gaussian kernel, and then, the Renyi's quadratic entropy is minimized using a gradient descent algorithm. By estimating Renyi's entropy over a short sliding window, an online training algorithm is also introduced. Moreover, for a linear equalizer, an orthogonality condition for the minimum entropy solution that leads to an alternative fixed-point iterative minimization method is derived. The performance of linear and nonlinear equalizers trained with entropy and mean square error (MSE) is compared. As expected, the results of training a linear equalizer are very similar for both criteria since, even if the input noise is non-Gaussian, the output filtered noise tends to be Gaussian. On the other hand, for nonlinear channels and using a multilayer perceptron (MLP) as the equalizer, differences between both criteria appear. Specifically, it is shown that the additional information used by the entropy criterion yields a faster convergence in comparison with the MSE     

Relationship Between Geometric Translations and TLS Estimation Bias in Bearings-Only Target Localization

This paper analyzes the effects of local coordinate translations and rotations on the bias and mean-squared error performance of the total least squares (TLS) bearings-only target localization algorithm. The TLS estimator was originally proposed to alleviate the severe bias problems associated with the traditional pseudolinear estimator. An interesting property of the TLS estimator, which is not shared by the pseudolinear estimator, is that its bias is sensitive to where the origin of the local Cartesian coordinates is placed. The paper provides a formal proof of this observation, discusses its implication on bias minimization, and proposes a simple and effective method for TLS bias reduction. The findings of the paper are illustrated with comprehensive simulation examples.     

A two-dimensional maximum entropy spectral estimator

Using ideas from one-dimensional maximum entropy spectral estimation a two-dimensional spectral estimator is derived by extrapolating the two-dimensional sampled autocorrelation (or covariance) function. The method used maximizes the entropy of a set of random variables. The extrapolation (or prediction) process under this maximum entropy condition is shown to correspond to the most random extension or equivalently to the maximization of the mean-square prediction error when the optimum predictor is used. The two-dimensional extrapolation must he terminated by the investigator. The Fourier transform of the extrapolated autocorrelation function is the two-dimensional spectral estimator. Using this method one can apply windowing prior to calculating the spectral estimate. A specific algorithm for estimating the two-dimensional spectrum is presented, and its computational complexity is estimated. The algorithm has been programmed and computer examples are presented.     

CPU散热器热分析与优化设计

利用CFD方法分析了平板直肋片散热器特性,通过多元线性回归建立了散热器换热和流动准则关系式,提供了散热器热阻和熵产率具体表达式;结合熵特性以本文提出的准则关系式采用约束条件的遗传优化算法对散热器结构进行多参数优化,优化结果与特性分析结论和相关文献吻合。     

Linear prediction, maximum flatness, maximum entropy, and ARpolyspectral estimation

The authors present a theoretical foundation for polyspectral estimation and modeling of non-Gaussian autoregressive (AR) processes which includes a new higher-order-statistics (HOS)-based linear prediction error filter and associated linear prediction polyspectral estimator, and a maximum higher order entropy polyspectral estimator, and considers the equivalences among these polyspectral estimators     

Convolutive blind source separation by minimizing mutual information between segments of signals

A method to perform convolutive blind source separation of super-Gaussian sources by minimizing the mutual information between segments of output signals is presented. The proposed approach is essentially an implementation of an idea previously proposed by Pham. The formulation of mutual information in the proposed criterion makes use of a nonparametric estimator of Renyi's /spl alpha/-entropy, which becomes Shannon's entropy in the limit as /spl alpha/ approaches 1. Since /spl alpha/ can be any number greater than 0, this produces a family of criteria having an infinite number of members. Interestingly, it appears that Shannon's entropy cannot be used for convolutive source separation with this type of estimator. In fact, only one value of /spl alpha/ appears to be appropriate, namely /spl alpha/=2, which corresponds to Renyi's quadratic entropy. Four experiments are included to show the efficacy of the proposed criterion.     

The additive causal part of Burg's maximum entropy estimator anddata consistency

The additive causal part of Burg's maximum entropy (ME) estimator spectrum is calculated in closed form. An immediate corollary for data consistency is shown     

Estimating the entropy of a signal with applications

We present a new estimator of the entropy of continuous signals. We model the unknown probability density of data in the form of an AR spectrum density and use regularized long-AR models to identify the AR parameters. We then derive both an analytical expression and a practical procedure for estimating the entropy from sample data. We indicate how to incorporate recursive and adaptive features in the procedure. We evaluate and compare the new estimator with other estimators based on histograms, kernel density models, and order statistics. Finally, we give several examples of applications. An adaptive version of our entropy estimator is applied to detection of law changes, blind deconvolution, and source separation     

Joint Phase and Timing Acquisition in Data-Transmission

This paper deals with the analysis of two different recursive methods forcarrierphaseand timing acquisition in synchronous data-transmission systems. The first method is a practical implementation of the maximum likelihood estimator of the phase and timing parameters; the other isbased on the minimization of the mean square error at the sampling times. In both cases the steady-state performance characteristics are evaluated andthe problem of the coupling between phase and timing adjustments is discussed in view of its effects upon stability and convergence rate of the synchronization algorithm. Finally, the signal design problem isaddressed with the objective of simultaneously attaining adjustment decoupling, jitter minimization and intersymbol interference reduction.     

High Accuracy Velo city Measurement Based on Keystone Transform Using Entropy Minimization

Velocity measurement is a basic task of radars. The target velocity is usually estimated according to the Doppler frequency shift. While traditional Doppler methods are unsuitable for high-speed targets, since the serious range migration between adjacent echoes causes phase wrapping. The serious range migration also inter-feres the coherent integration to improve the accuracy of the velocity estimation. A velocity measurement method based on Keystone transform using entropy minimization is studied to solve this problem. This method applies Key-stone transform to the echo to calculate the ambiguity de-gree with the help of entropy minimization. The proposed algorithm estimates the ambiguity degree with no error at a wider range of SNR than the traditional method. The ambiguous Doppler frequency is obtained according to the slow time. Theoretical analyses and simulations show that this method has very high precision.     

Conditional distribution learning with neural networks and itsapplication to channel equalization

We present a conditional distribution learning formulation for real-time signal processing with neural networks based on an extension of maximum likelihood theory-partial likelihood (PL) estimation-which allows for (i) dependent observations and (ii) sequential processing. For a general neural network conditional distribution model, we establish a fundamental information-theoretic connection, the equivalence of maximum PL estimation, and accumulated relative entropy (ARE) minimization, and obtain large sample properties of PL for the general case of dependent observations. As an example, the binary case with the sigmoidal perceptron as the probability model is presented. It is shown that the single and multilayer perceptron (MLP) models satisfy conditions for the equivalence of the two cost functions: ARE and negative log partial likelihood. The practical issue of their gradient descent minimization is then studied within the well-formed cost functions framework. It is shown that these are well-formed cost functions for networks without hidden units; hence, their gradient descent minimization is guaranteed to converge to a solution if one exists on such networks. The formulation is applied to adaptive channel equalization, and simulation results are presented to show the ability of the least relative entropy equalizer to realize complex decision boundaries and to recover during training from convergence at the wrong extreme in cases where the mean square error-based MLP equalizer cannot     

Approximate all nearest neighbor search for high dimensional entropy estimation for image registration

Information theoretic criteria such as mutual information are often used as similarity measures for inter-modality image registration. For better performance, it is useful to consider vector-valued pixel features. However, this leads to the task of estimating entropy in medium to high dimensional spaces, for which standard histogram entropy estimator is not usable. We have therefore previously proposed to use a nearest neighbor-based Kozachenko-Leonenko (KL) entropy estimator.Here we address the issue of determining a suitable all nearest neighbor (NN) search algorithm for this relatively specific task.We evaluate several well-known state-of-the-art standard algorithms based on k-d trees (FLANN), balanced box decomposition (BBD) trees (ANN), and locality sensitive hashing (LSH), using publicly available implementations. In addition, we present our own method, which is based on k-d trees with several enhancements and is tailored for this particular application.We conclude that all tree-based methods perform acceptably well, with our method being the fastest and most suitable for the all-NN search task needed by the KL estimator on image data, while the ANN and especially FLANN methods being most often the fastest on other types of data. On the other hand, LSH is found the least suitable, with the brute force search being the slowest.     

Generalization of Maximum A Posteriori Amplitude Estimator Under Speech Presence Uncertainty for Speech Enhancement

In this paper, we focus on the estimation-based frequency-domain speech enhancement methods under speech presence uncertainty. Through the minimization of an average risk function, a generalization of maximum a posteriori spectral amplitude estimator is derived. By adjusting the cost parameters, we can control the error caused by noise falsely detected as speech. Our experimental results show that the proposed system can be a simple alternative to Abramson’s simultaneous detection and estimation approach for speech enhancement since it involves merely estimation under speech presence uncertainty and does not require any detector. Moreover, the proposed estimator takes advantage of a more straightforward implementation, since there is no need for the computation of Bessel functions.     

Volume image registration by cross-entropy optimization

Cross-entropy (CE), an information-theoretic measure, quantifies the difference between two probability density functions. This measure is applied to volume image registration. When a good prior estimation of the joint distribution of the voxel values of two images in registration is available, the CE can be minimized to find an optimal registration. If such a prior estimation is not available, one seeks the registration which gives a joint distribution different from unlikely ones as much as possible, i.e., the CE is maximized to find an optimal registration. When the unlikely distribution is a uniform one, CE maximization reduces to joint entropy minimization; when the unlikely distribution is proportional to one of the marginal distributions, it reduces to conditional entropy minimization; when the unlikely distribution is the product of two marginal distributions, it degenerates to mutual-information maximization. These different CEs are added together and are used as criteria for image registration. The accuracy and robustness of this new approach are tested and compared using a likely joint distribution and various unlikely joint distributions and their combinations.     

An asymptotically unbiased estimator for bearings-only and Doppler-bearing target motion analysis

Bearings-only (BO) and Doppler-bearing (DB) target motion analysis (TMA) attempt to obtain a target trajectory based on bearings and on Doppler and bearing measurements, respectively, from an observer to the target. The BO-TMA and DB-TMA problems are nontrivial because the measurement equations are nonlinearly related to the target location parameters. The pseudolinear formulation provides a linear estimator solution, but the resulting location estimate is biased. The instrumental variable method and the numerical maximum likelihood approach can eliminate the bias. Their convergence behavior, however, is not easy to control. This paper proposes an asymptotically unbiased estimator of the tracking problem. The proposed method applies least squares minimization on the pseudolinear equations with a quadratic constraint on the unknown parameters. The resulting estimator is shown to be solving the generalized eigenvalue problem. The proposed solution does not require initial guesses and does not have convergence problems. Sequential forms of the proposed algorithms for both BO-TMA and DB-TMA are derived. The sequential algorithms improve the estimation accuracy as a new measurement arrives and do not require generalized eigenvalue decomposition for solution update. The proposed estimator achieves the Cramer-Rao Lower Bound (CRLB) asymptotically for Gaussian noise before the thresholding effect occurs.