Cumulative residual entropy: a new measure of information

In this paper, we use the cumulative distribution of a random variable to define its information content and thereby develop an alternative measure of uncertainty that extends Shannon entropy to random variables with continuous distributions. We call this measure cumulative residual entropy (CRE). The salient features of CRE are as follows: 1) it is more general than the Shannon entropy in that its definition is valid in the continuous and discrete domains, 2) it possesses more general mathematical properties than the Shannon entropy, and 3) it can be easily computed from sample data and these computations asymptotically converge to the true values. The properties of CRE and a precise formula relating CRE and Shannon entropy are given in the paper. Finally, we present some applications of CRE to reliability engineering and computer vision.     

Minimum entropy quantizers and permutation codes

Amplitude quantization and permutation encoding are two approaches to efficient digitization of analog data. It has been proven that they are equivalent in the sense that their optimum rate versus distortion performances are identical. Reviews of the aforementioned results and of work performed in the interim by several investigators are presented. Equations which must be satisfied by the thresholds of the minimum entropy quantizer that achieves a prescribed meanrth power distortion are derived, and an iterative procedure for solving them is developed. It is shown that these equations often have many families of solutions. In the case of the Laplacian distribution, for which we had previously shown that quantizers with uniformly spaced thresholds satisfy the equations whenr=2, other families of solutions with nonuniform spacing are exhibited. What had appeared to be a discrepancy between the performances of optimum permutation codes and minimum entropy quantizers is resolved by the resulting optimum quantizers, which span all entropy rates from zero to infinity.     

Minimum spatial entropy threshold selection

Many image grey level thresholding methods based on the theory of maximum entropy have been proposed in the past. However, the exact definition of what is meant by `image entropy' has varied considerably, with measures based on the image itself, its histogram or other related distributions being proposed. Most of these ignore the spatial component of the image, assuming pixels within the image to be independent of each other. While this is convenient, it is counterintuitive. A recent measure of spatial entropy, which attempts to include the spatial information inherent in the image, has been used here as a criterion function for threshold selection. Results indicate an improvement over methods that ignore this information     

Minimum relative entropy enhancement of FFT spectra

The typical method for obtaining a power spectrum from values of the correlation function is through the FFT (fast Fourier transform). A novel approach to improving the FFT spectrum when it is corrupted by noise is presented. The method minimises a relative entropy functional involving spectral densities and requires knowledge only of the variance of the noise that is corrupting the FFT spectrum.<>     

Minimum entropy approach for carrier frequency recovery

This paper introduces a new carrier frequency recovery approach. It can be applied before modulation classification and/or demodulation of the M-ary PSK signals. It relies on the entropy of the instantaneous phase probability density function, and uses the fact that it reaches minimum when the receiver is fine-tuned to the unknown carrier frequency. This estimator is applicable to algorithms requiring high accuracy without any a priori knowledge concerning modulation scheme, signal contents (bit-stream), or its timing parameters. Simulation results have proved the robustness of the algorithm: for low carrier to noise ratios (CNR), corresponding variances are proportional and close to Cramer-Rao lower bounds (CRLB). For CNR greater than 20 dB, they are constant and limited by the resolution of the algorithm.     

Minimum information spectral analysis

The problem of obtaining an adequate spectral estimate based on the information provided by only a finite number of observations regarded as an incomplete realization of a discrete wide sense stationary stochastic process is considered. From an a priori choice in favour of either the univariate autoregressive (AR), moving average (MA) or the mixed ARMA parameterization and the availability of some estimated autocovariance lags, both recursive and non-recursive estimators are obtained by maximizing the spectral entropy. These estimators preserve minimum information subject to the unknown autocovariance lags. Moreover, the approach of maximizing the spectral entropy is shown to be equivalent to the application of a rectangular window to the cepstrum.     

Minimum entropy of error estimation for discrete random variables

The principle of minimum entropy of error estimation (MEEE) is formulated for discrete random variables. In the case when the random variable to be estimated is binary, we show that the MEEE is given by a Neyman-Pearson-type strictly monotonous test. In addition, the asymptotic behavior of the error probabilities is proved to be equivalent to that of the Bayesian test     

用熵与互信息探究信息传输原理

根据信息及其传输的实质建立信息传输的物理模型。用熵与互信息研究信息传输的有效性和可靠性，通过编码和编码定理得到一些重要的性能界限。最后建立了一个数学模型，进一步揭示信息传输原理及其核心理论。     

Minimum entropy regularization in frequency-wavenumber migration to localize subsurface objects

Optimized versions of frequency-wavenumber (F-K) migration methods are introduced to better focus ground-penetrating radar (GPR) data in applications of shallow subsurface object localization, e.g., landmine remediation. Migration methods are based on the wave equation and operate by backpropagating the received data into the earth so as to localize buried objects. Traditional F-K migration is based on an underlying assumption that the wavefields propagate in a homogeneous medium. The presence of a rough air-ground interface in the GPR case degrades the localization ability. To overcome this problem in the context of the F-K algorithm, we introduce lateral variations in the velocity of waves in the medium. An optimization approach is employed to choose that velocity function that results in a well-focused image where an entropy-like criterion is used to quantify the notion of focus. Extension of the basic method to lossy medium is also described. The utility of these techniques is demonstrated using field data from a number of GPR systems.     

基于高斯混合及相对熵测度的图像分割

针对图像分割问题,结合高斯混合模型与信息论中的相对熵测度概念,提出一种新的图像阈值化方法。在提出方法中把图像阈值化问题看成是两个概率向量之间的匹配问题,因此首先用高斯混合模型去拟合图像直方图的灰度级分布,然后用相对熵测度去度量拟合分布与图像原灰度级分布之间的差异,并把该度量作为图像阈值化的准则函数。在对图像实施分割时,通过在图像灰度级范围中求取所定义的准则函数的最小值获得最佳阈值。在NDT、SAR及红外图像上的分割实验中用提出方法与传统及最新的图像阈值化方法进行比较,结果表明提出方法获得的结果要优于相比较方法获得的分割结果,因此提出方法是一种有效的图像分割方法。     

交叉熵约束的红外图像最小错误阈值分割

针对目标和背景具有相似统计分布的红外图像,经典阈值分割方法仅以某种形式的方差或熵作为准则,未考虑图像的实际特性,分割效果不甚理想。为此,提出了一种基于交叉熵约束的红外图像最小错误阈值分割新方法。首先,引入交叉熵来度量目标和背景统计分布的相似性,交叉熵越小表明分布越相似;然后在交叉熵小于一定值的条件下使分类错误达到最小。交叉熵的约束保证了分割过程适应红外图像实际特性,分类错误最小确保了分割效果的有效性。该方法原理清晰、参数设置简单,在一系列实际图像上的实验结果表明,与现有几种经典阈值分割方法相比,文中方法有效提高了目标和背景具有相似统计分布的红外图像的阈值分割准确率。     

开放互联网环境基于信息熵的信息传播影响力计算方法

研究了开放互联网环境下衡量信息传播的影响力,提出了一种基于最大熵理论的信息传播影响力计算方法。首先,对信息传播影响力所涉及的要素进行建模;然后,以香农信息熵理论为基础,根据“奥卡姆剃刀原理”选取信息影响力关联指标;最后,基于信息熵理论给出了信息传播影响力的计算方法。对学术论文引文和网络舆情事件的传播影响力进行了分析,验证了所提方法的有效性,所提方法对探究信息在开放互联网环境中的传播影响力以及对受众的影响程度具有重要的现实意义。     

Minimum error entropy estimation and entropic prediction filtering: An optimal predictive coding scheme

The minimum error entropy (MEE) estimation problem on discrete, finite, or countably infinite ensembles of arbitrary statistical nature is considered. A combinatorial approach is taken to solve the minimization problem. The solution is obtained in algorithmic form. The numerical complexity of the MEE algorithm is analyzed. An entropic prediction filtering (EPF) coding scheme that utilizes the MEE algorithm to minimize the per symbol error entropy is introduced. Simulation has shown that the EPF can effectively be used to reduce the average codeword length required to encode data. Other applications of the MEE algorithm in signal interpolation and extrapolation are also discussed.     

基于最小熵的SAR干扰图像配准与评估研究

噪声干扰前后SAR图像的有效配准是判断其受干扰程度、计算图像相关性的前提,也是开展SAR对抗效果评估的基本环节。针对SAR对抗试验中噪声干扰前后机载平台航次一致性难以精确保证的问题,提出了一种基于最小熵子域相关法的SAR干扰图像配准策略。首先,按照一定的搜索方法在干扰前后的SAR图像中进行相关子域选取;然后,按照最小熵准则进行SAR干扰图像配准;最后,分析推导了配准性能与受干扰程度之间的关系。仿真实例验证了该配准策略的有效性。     

An information entropy approach to the small-lot concept

Just-in-time and total quality management advocate the use of small lots in production, as they yield higher throughput, better quality, lower response time, less operating expenses, better due date performance, and less work in process. Implementation of the small-lot concept is often resisted by MIS managers and production people who feel that the larger number of lots-the apparently inevitable result of reducing lot size, will entail more information and stronger information technology. By developing a normative model, based on the entropy measurement, this paper claims that the move towards smaller lots implies less information needs. Theorems concerning the relationship between quantity of information and lot size are proved, showing that the new directions that manufacturing is taking entail less information needs     

Estimation of the entropy and information of absolutely continuousrandom variables

A method is proposed for estimating the entropy and the mutual information of absolutely continuous random vectors, and an upper bound of the mean risks for the proposed estimators under strong mixing conditions is given     

基于信息熵的船舶航行与区域监控方法

针对内河与沿海日益复杂及巨量的船舶目标活动带来的监管挑战,开展了船舶AIS信息应用研究。通过数据清洗、目标编批实现AIS船舶目标的初步整编和区域划分,并基于信息熵方法评估船舶目标航行状态,进而提供区域目标密度、区域航行风险评估。研究成果有助于提高目标与区域精细管理能力,降低水面交通风险。     

无线传感器网络延时测量数据的信息熵与不确定性

无线传感器网络的时间同步是无线传感器网络的一个重要的研究方向.本文利用时钟偏移量误差的规律性,提出基于指数时延的无线传感器网络时钟同步算法.在DMTS算法的基础上引入指数时延得到了发送者的同步模型.针对无线传感器网络延迟测量数据处理中掺杂的主观因素不能准确反映客观现实的问题,采用最大熵方法,根据测量数据求取被测量的概率...     

Image fusion metric based on mutual information and Tsallis entropy

A novel image fusion performance metric using mutual information is proposed. The metric is based on Tsallis entropy, which is a one-parameter generalisation of Shannon entropy. Experimental results have confirmed that the proposed metric outperforms the standard MI metric by correlating better with the subjective quality of fused images.     

基于背景差分和信息熵的运动目标检测算法

提出一种目标检测算法,首先选取视频的第一帧作为背景帧,运用加权累加图像方法更新背景图像,背景图像的更新速率通过相邻帧的差分结果决定,再运用背景差分算法提取出运动目标。计算运动目标的区域的熵值,通过熵值判断出特征目标。实验结果表明,该算法简单,稳定性好,能够较好解决动态背景的问题并且检测出特定目标。