A simple information theoretic proof of the maximum entropyproperty of some Gaussian random fields

A well known result of Burg (1967) and Kunsch (1981) identifies a Gaussian Markov random field with autocovariances specified on a finite part L of the n-dimensional integer lattice, as the random field with maximum entropy among all random fields with same autocovariance values on L. A simple information theoretic proof of a version of the maximum entropy theorem for random fields in n dimensions is presented in the special case that the given autocovariances are compatible with a unilateral autoregressive process.     

Maximum entropy and conditional probability

It is well-known that maximum entropy distributions, subject to appropriate moment constraints, arise in physics and mathematics. In an attempt to find a physical reason for the appearance of maximum entropy distributions, the following theorem is offered. The conditional distribution ofX_{l}given the empirical observation(1/n)sum^{n}_{i}=_{l}h(X_{i})=alpha, whereX_{1},X_{2}, cdotsare independent identically distributed random variables with common densitygconverges tof_{lambda}(x)=e^{lambda^{t}h(X)}g(x)(Suitably normalized), wherelambdais chosen to satisfyint f_{lambda}(x)h(x)dx= alpha. Thus the conditional distribution of a given random variableXis the (normalized) product of the maximum entropy distribution and the initial distribution. This distribution is the maximum entropy distribution whengis uniform. The proof of this and related results relies heavily on the work of Zabell and Lanford.     

Eigenvector properties of Toeplitz matrices and their application to spectral analysis of time series

This paper presents a proof for an important eigenvector property of Toeplitz matrices. The property, a variant of Caratheodory's theorem, is that the zeroes of the polynomials formed from eigenvectors with unique maximum/minimum eigenvalues of a positive definite Toeplitz matrix are distinct and lie on the unit circle. The proof is based on simple matrix-theoretical results and offers an insight that is lacking in existing proofs. A representation form for Toeplitz matrices is obtained and this form is used for justification of Pisarenko's spectral analysis based on eigenvectors/eigenvalues of the covariance matrix. The theoretical groundwork developed for the proof of the eigenvector property is then used to compare Pisarenko's null eigenvector technique with Burg's maximum entropy analysis, and arguments are presented in favor of the former technique.     

Entropy and the timing capacity of discrete queues

Queueing systems which map Poisson input processes to Poisson output processes have been well-studied in classical queueing theory. This paper considers two discrete-time queues whose analogs in continuous-time possess the Poisson-in-Poisson-out property. It is shown that when packets arriving according to an arbitrary ergodic stationary arrival process are passed through these queueing systems, the corresponding departure process has an entropy rate no less (some times strictly more) than the entropy rate of the arrival process. Some useful by-products are discrete-time versions of: (i) a proof of the celebrated Burke's (1956) theorem, (ii) a proof of the uniqueness, amongst renewal inputs, of the Poisson process as a fixed point for exponential server queues proposed by Anantharam (1993), and (iii) connections with the timing capacity of queues described by Anantharam and Verdu (1996).     

Maximum entropy spectral analysis and ARMA processes (Corresp.)

The maximum entropy spectral analysis for discrete-time stationary processes was first proposed by J. P. Burg. He showed that if a finite number of covariance lag values of a stationary process are known, then an autoregressive (AR) process with the given autocorrelation values best fits the given constraints in the sense of maximizing thc differential entropy rate of the model. A more general type of prior knowledge of the process is considered, and it is shown that the maximum entropy method, subject to our constraints, is equivalent to fitting a mixed autoregressive moving average (ARMA) model.     

Image reconstruction by conditional entropy maximisation for PET system

The authors show that the conditional entropy maximisation algorithm is a generalised version of the maximum likelihood algorithm for positron emission tomography (PET). Promising properties of the conditional entropy maximisation algorithm are as follows: an assumption is made that the entropy of the information content of the data should be maximised; it is a consistent way of selecting an image from the very many images that fit the measurement data; this approach takes care of the positivity of the reconstructed image pixels, since entropy does not exist for negative image pixel values; and inclusion of prior distribution knowledge in the reconstruction process is possible. Simulated experiments performed on a PET system have shown that the quality of the reconstructed image using the entropy maximisation method is good. A Gibbs distribution is used to incorporate prior knowledge into the reconstruction process. The mean squared error (MSE) of the reconstructed images shows a sharp new dip, confirming improved image reconstruction. The entropy maximisation method is an alternative approach to maximum likelihood (ML) and maximum a posteriori (MAP) methodologies.     

Extension to the maximum entropy method

The maximum entropy method provides an estimate of the power spectral density which maximizes the entropy of a stationary random process from the firstNlags of the autocorrelation function. The method is extended to accommodate weighted errors in the measured autocorrelation function.     

最大熵方法及其在自然语言处理中的应用研究

本文研究当前的自然语言处理状况和最大熵方法的发展,着重探讨最大熵方法在自然语言处理的过程中的应用范围,希望能为相关人士提供有效参考。     

基于安全熵的间接非授权行为分析理论

针对系统中存在间接非授权访问可能性的量化分析和证明问题,提出了一种基于安全熵的量化分析理论.首先,结合信息论有关知识引入安全熵概念,提出系统对间接非授权访问行为响应的不确定性计算方法;然后,基于安全熵提出了系统的间接安全性定理,作为判断系统是否可能存在间接非授权访问的依据;最后,应用该方法对经典安全模型进行了量化分析,验证了该方法的实用性.结果证明该方法适用于系统或访问控制模型对间接非授权访问的防护能力评估和证明.     

Information of deterministic functions and quantum information of non probabilistic matrices A unified approach and new results

This paper provides new results on the entropy of functions on the one hand, and exhibits a unified approach to entropy of functions and quantum entropy of matrices with or without probability. The entropy of continuously differentiable functions is extended to stair-wise functions, a measure of relative information between two functions is obtained, which is fully consistent with Kullback cross-entropy, Renyi cross-entropy and Fisher information. The theory is then applied to stochastic processes to yield some concepts of random geometrical entropies defined on path space, which are related to fractal dimension in the special case when the process is a fractional Brownian. Then it shows how one can obtain Shannon entropy of random variables by combining the maximum entropy principle with Hartley entropy. Lastly quantum entropy of non probabilistic matrices (extension of Von Neumann quantum mechanical entropy) is derived as a consequence of Shannon entropy of random variables.     

Extension to the maximum entropy method II

The maximum entropy method provides an estimate of the power spectral density which maximizes the entropy of a stationary random process from the firstNlags of the autocorrelation function. The method is applied to cases where some of the firstNautocorrelation function lags are unknown.     

线性退化二值图象的最大熵复原法

本文主要用最大熵方法对线性退化二值图象的复原进行了研究。利用原始图象的二值特性,提出了二值约束的最大熵复原方法,并对其解的存在性和唯一性进行了论述,对双约束最大熵的求解问题给出了算法;运用最大有界熵概念,提出了二值约束的最大有界熵复原法。文中将上述复原法同维纳滤波法和最大熵法进行了比较,实验结果表明,用二值约束的最大(有界)熵法复原线性退化图象可以提高复原的质量。     

Notes on maximum-entropy processing (Corresp.)

Maximum-entropy processing is a method for computing the power density spectrum from the firstNlags of the autocorrelation function. Unlike the discrete Fourier transform, maximum-entropy processing does not assume that the other lag values are zero. Instead, one mathematically ensures that the fewest possible assumptions about unmeasured data are made by choosing the spectrum that maximizes the entropy for the process. The use of the maximum entropy approach to spectral analysis was introduced by Burg [1]. In this correspondence, the authors derive the maximum-entropy spectrum by obtaining a spectrum that is forced to maximize the entropy of a stationary random process.     

MAXIMUM ENTROPY RESTORATION METHOD OF LINEARLY DEGRADED BINARY IMAGE

This paper investigates the maximum entropy restoration of blurred binary image.In concerning with the binary property of image,a new maximum entropy restoration methodwith binary constraint is proposed.The properties of existence and uniqueness of solution arediscussed.The problem of maximum of entropy with two constraints is solved and the corre-sponding algorithm is given.In this paper,the maximum bounded entropy principle is employedconcerning the prior knowledge of binary image,and the maximum bounded entropy restora-tion method with binary constraint is put forward.The proposes methods,Wiener filter(WF)restoration method and maximum entropy restoration method are compared.The experimen-tal results show that the maximum entropy restoration method and maximum bounded entropyrestoration method with binary constraint can improve the quality of restored image.     

The extended MENT algorithm: a maximum entropy type algorithm usingprior knowledge for computerized tomography

Unlike MENT (maximum entropy algorithm), the extended MENT algorithm can process prior information and deal with incomplete projections or limited angle data. The reconstruction problem is formulated for solving linear systems involving the Fredholm integral equation. To develop the extended MENT algorithm, maximum entropy is substituted by a more general optimization criterion, that of minimizing the discriminatory function. The a priori knowledge of the shape of the object is easily incorporated in the algorithm by using the discriminatory function. Useful mathematical properties that make the discriminatory function attractive are derived. The sensitivity of the minimum discriminatory solution is derived to determine the characteristics of the noise in the reconstructed images. The extended MENT algorithm is developed for a parallel geometry, and its convergence properties are given. Its image processing performance is better than that for other maximum entropy algorithms such as multiplicative algebraic reconstruction techniques (MART) or more standard methods such as ART and the convolution backprojection     

Discrete minimum entropy quantization

We present a simple lower bound on the entropy of a quantized signal and a method for determining the minimum entropy quantization, subject to a maximum distortion, for a discrete memoryless random process. This quantization allows more efficient use of variable-order models for compression of images and sound.     

Maxentropic Markov chains (Corresp.)

The Markov chain that has maximum entropy for given first and second moments is determined. The solution provides a discrete analog to the continuous Gauss-Markov process.     

A two-dimensional maximum entropy spectral estimator

Using ideas from one-dimensional maximum entropy spectral estimation a two-dimensional spectral estimator is derived by extrapolating the two-dimensional sampled autocorrelation (or covariance) function. The method used maximizes the entropy of a set of random variables. The extrapolation (or prediction) process under this maximum entropy condition is shown to correspond to the most random extension or equivalently to the maximization of the mean-square prediction error when the optimum predictor is used. The two-dimensional extrapolation must he terminated by the investigator. The Fourier transform of the extrapolated autocorrelation function is the two-dimensional spectral estimator. Using this method one can apply windowing prior to calculating the spectral estimate. A specific algorithm for estimating the two-dimensional spectrum is presented, and its computational complexity is estimated. The algorithm has been programmed and computer examples are presented.     

CMOS compatible bulk micromachined silicon piezoresistive accelerometer with low off-axis sensitivity

The CMOS compatible bulk micromachined piezoresistive accelerometer presented in this paper consists of four flexures supporting a proof mass. Four pairs of boron-diffused piezoresistors are located at maximum stress points on the flexures near the proof mass and frame ends. Because of the opposite nature of stress at the two ends, these piezoresistors can be connected to form a Wheatstone bridge such that the off-axis responses are practically cancelled while the on-axis (along perpendicular to proof mass) response is maximized. The device is simulated using CoventorWare. In the fabrication process, dual-doped TMAH solution is used for wet anisotropic etching. The novelty of this etching process is that the bulk micromachining can be performed after aluminum metallization. The etched surface is also smooth. The fabrication is thus CMOS compatible. The accelerometer exhibits good linearity over 0-10 g.     

粒子滤波算法在多传感器测量中的应用

目标跟踪是粒子滤波算法在处理非线性问题的一种典型应用,但由于在线处理能力或传输条件的限制,实际应用中往往无法对多个传感器数据同时处理。据此,给出了一种基于多传感器选优的粒子滤波算法。假设每个时刻可以处理一个测量数据,该算法先采用加权的概率密度函数来评价每个传感器获得的测量值,并用粒子滤波对概率密度函数的加权进行实时更新,基于最大熵标准来选取最优测量数据进行处理。同时,最大熵标准保证了最优似然函数分布最宽,从而缓解粒子衰竭问题。通过数值仿真实验证明,该算法可以选择最优观测数据进行处理,有效降低多传感器测量中粒子滤波在线实时处理性能的要求,也较好地缓解了粒子滤波的"衰竭"问题。