Continuously Differentiable Sample-Spacing Entropy Estimation

The insufficiency of using only second-order statistics and premise of exploiting higher order statistics of the data has been well understood, and more advanced objectives including higher order statistics, especially those stemming from information theory, such as error entropy minimization, are now being studied and applied in many contexts of machine learning and signal processing. In the adaptive system training context, the main drawback of utilizing output error entropy as compared to correlation-estimation-based second-order statistics is the computational load of the entropy estimation, which is usually obtained via a plug-in kernel estimator. Sample-spacing estimates offer computationally inexpensive entropy estimators; however, resulting estimates are not differentiable, hence, not suitable for gradient-based adaptation. In this brief paper, we propose a nonparametric entropy estimator that captures the desirable properties of both approaches. The resulting estimator yields continuously differentiable estimates with a computational complexity at the order of those of the sample-spacing techniques. The proposed estimator is compared with the kernel density estimation (KDE)-based entropy estimator in the supervised neural network training framework with computation time and performance comparisons.      

Entropy based robust estimator and its application to line-based mapping

This paper presents a robust mapping algorithm for an application in autonomous robots. The method is inspired by the notion of entropy from information theory. A kernel density estimator is adopted to estimate the appearance probability of samples directly from the data. An Entropy Based Robust (EBR) estimator is then designed that selects the most reliable inliers of the line segments. The inliers maintained by the entropy filter are those samples that carry more information. Hence, the parameters extracted from EBR estimator are accurate and robust to the outliers. The performance of the EBR estimator is illustrated by comparing the results with the performance of three other estimators via simulated and real data.     

Entropy estimation in Turing's perspective

A new nonparametric estimator of Shannon's entropy on a countable alphabet is proposed and analyzed against the well-known plug-in estimator. The proposed estimator is developed based on Turing's formula, which recovers distributional characteristics on the subset of the alphabet not covered by a size-n sample. The fundamental switch in perspective brings about substantial gain in estimation accuracy for every distribution with finite entropy. In general, a uniform variance upper bound is established for the entire class of distributions with finite entropy that decays at a rate of O(ln(n)/n) compared to O([ln(n)]2/n) for the plug-in. In a wide range of subclasses, the variance of the proposed estimator converges at a rate of O(1/n), and this rate of convergence carries over to the convergence rates in mean squared errors in many subclasses. Specifically, for any finite alphabet, the proposed estimator has a bias decaying exponentially in n. Several new bias-adjusted estimators are also discussed.     

Linear metrics for entropy estimation

An entropy estimator constructed with respect to specially selected metrics is studied. It is shown that the estimator converges almost everywhere and the decrease in its variance is of an power order. For symmetric Bernoulli measures, the bias of the estimator is found.     

Density estimation with stagewise optimization of the empirical risk

We consider multivariate density estimation with identically distributed observations. We study a density estimator which is a convex combination of functions in a dictionary and the convex combination is chosen by minimizing the L ₂ empirical risk in a stagewise manner. We derive the convergence rates of the estimator when the estimated density belongs to the L ₂ closure of the convex hull of a class of functions which satisfies entropy conditions. The L ₂ closure of a convex hull is a large non-parametric class but under suitable entropy conditions the convergence rates of the estimator do not depend on the dimension, and density estimation is feasible also in high dimensional cases. The variance of the estimator does not increase when the number of components of the estimator increases. Instead, we control the bias-variance trade-off by the choice of the dictionary from which the components are chosen. Editor: Nicolo Cesa-Bianchi     

Estimating the entropy rate of spike trains via Lempel-Ziv complexity

Normalized Lempel-Ziv complexity, which measures the generation rate of new patterns along a digital sequence, is closely related to such important source properties as entropy and compression ratio, but, in contrast to these, it is a property of individual sequences. In this article, we propose to exploit this concept to estimate (or, at least, to bound from below) the entropy of neural discharges (spike trains). The main advantages of this method include fast convergence of the estimator (as supported by numerical simulation) and the fact that there is no need to know the probability law of the process generating the signal. Furthermore, we present numerical and experimental comparisons of the new method against the standard method based on word frequencies, providing evidence that this new approach is an alternative entropy estimator for binned spike trains.     

On minimal error entropy stochastic approximation

Information-theoretic concepts are developed and employed to obtain conditions for a minimax error entropy stochastic approximation algorithm to estimate the state of a non-linear discrete time system baaed on noisy linear measurements of the state. Two recursive suboptimal error entropy estimation procedures are presented along with an upper bound formula for the resulting error entropy. A simple example is utilized to compare the optimal and suboptimal error entropy estimators and the minimum mean Square error linear estimator.     

Novel hardware processing unit for dynamic on-line entropy estimation of discrete time information

This paper presents a novel and generic hardware processing unit that estimates the information entropy in a dynamic and on-line fashion with a simple architecture that can be easily scaled. This architecture does not require precomputations, change of domain at the input signal, or complex schemes of computation. Results show that the proposed FPGA implementation of the dynamic entropy estimator is highly efficient as a stand-alone system. Speed performance of the system is 3 orders of magnitude higher than its implementation counterpart in software with a maximum error of 1.5%. Compared with other hardware structures, the proposed architecture is able to process twice the information than a LUT-based entropy estimator during a time unit. Results also show that the proposed dynamic hardware processing unit is highly accurate carrying out standard tasks such as computing the information content in a discrete data set, or nonstandard tasks as detecting failures in induction motors.     

Stochastic observability in network state estimation and control

A hidden Markov model for the traffic congestion control problem in transmission control protocol (TCP) networks is developed, and the question of observability of this system is posed. Of specific interest are the dependence of observability on the congestion control law and the interaction between observability ideas and the effectiveness of feedback control. Analysis proceeds with a survey of observability concepts and an extension of some available definitions for linear and nonlinear stochastic systems. The key idea is to link the improvement of state estimator performance to the conditioning on the output data sequence. The observability development proceeds from linear deterministic systems to linear Gaussian systems, nonlinear systems, etc., with backwards compatibility to deterministic ideas. The principal concepts relate to the entropy decrease of scalar functions of the state, which in the linear case are describable in terms of covariance matrices. A feature of nonlinear systems is that the estimator properties may affect the closed-loop control performance. Results are derived linking stochastic reconstructibility to strict improvement of the optimal closed-loop control performance over open-loop control for the hidden Markov model. The entropy provides a means to quantify and thus order simulation results for a simplified TCP network. Motivated by the link between feedback control and reconstructibility, the entropy formulation is also explored as a means to discriminate between different control strategies for improving estimator performance. This approach has connections to dual-adaptive control ideas, where the control has the simultaneous and opposing goals of regulating the system and of exciting the system to prevent estimator divergence.     

Fuzzy entropy in fuzzy weighted linear regression model under linear restrictions with simulation study

In fuzzy set theory, it is well known that a fuzzy number can be uniquely determined through its position and entropy. Hence, by using the concept of fuzzy entropy the estimators of the fuzzy regression coefficients may be estimated. In the present communication, a fuzzy linear regression (FLR) model with some restrictions in the form of prior information has been considered. The estimators of regression coefficients have been obtained with the help of fuzzy entropy for the restricted/unrestricted FLR model by assigning some weights in the distance function. Some numerical examples have also been provided in order to illustrate the proposed model along with the obtained weighted estimators. Further, in order to compare the performance of unrestricted estimator and restricted estimator, a simulation study has been conducted by using two fundamental criteria of dominance – mean squared error matrix (MSEM) and absolute bias.     

Beyond second-order statistics for learning: A pairwise interaction model for entropy estimation

Second order statistics have formed the basisof learning and adaptation due to its appealand analytical simplicity. On the other hand,in many realistic engineering problemsrequiring adaptive solutions, it is notsufficient to consider only the second orderstatistics of the underlying distributions. Entropy, being the average information contentof a distribution, is a better-suited criterionfor adaptation purposes, since it allows thedesigner to manipulate the information contentof the signals rather than merely their power. This paper introduces a nonparametric estimatorof Renyi's entropy, which can be utilized inany adaptation scenario where entropy plays arole. This nonparametric estimator leads to aninteresting analogy between learning andinteracting particles in a potential field. Itturns out that learning by second orderstatistics is a special case of thisinteraction model for learning. We investigatethe mathematical properties of thisnonparametric entropy estimator, provide batchand stochastic gradient expressions foroff-line and on-line adaptation, and illustratethe performance of the corresponding algorithmsin examples of supervised and unsupervisedtraining, including time-series prediction andICA.     

Asymptotic form of the bias of entropy estimator for Bernoulli measures

The exact asymptotic form for the bias of entropy estimator [8] for Bernoulli measures is found.     

Record value based on intuitionistic fuzzy random variables

This paper extends some basic concepts associated to record value based on intuitionistic fuzzy random variables. In this approach, αβ-values of intuituinistic fuzzy numbers are employed to construct intuitionistic fuzzy cumulative distribution function and its common estimator, an extended entropy and its estimator, intuitionistic fuzzy (upper) record value and its common estimator. Main property of the proposed concepts include large sample properties which are investigated in the space of intuitionistic fuzzy numbers. Some numerical examples are also illustrated to clarify the concepts and methods.     

面向小目标图像的快速核密度估计图像阈值分割算法

针对当前小目标图像阈值分割研究工作面临的难题,提出了快速核密 度估计图像阈值分割新方法.首先给出了基于加权核密度估计器的概率计算模 型,通过引入二阶Renyi熵作为阈值选取准则,提出了基于核密度估计的图像阈 值分割算法 (Kernel density estimator based image thresholding algorithm, KDET), 然后通过引入快速压缩集密度估计 (Fast reduced set density estimator, FRSDE)技术,得到核密度估计的 稀疏权系数表示形式,提出快速核密度估计图像阈值分割算法fastKDET,并从 理论上对相关性质进行了深入探讨.实验表明,本文算法对小目标图像 阈值分割问题具有更广泛的适应性,并且对参数变化不敏感.     

参数估计定理

空间信息论是关于雷达等目标参数估计系统信息获取一般规律的基本理论,这里的空间信息指被测目标相对于雷达的距离、方向和散射信息。参数估计定理是空间信息论的重要组成部分,本文提出参数估计定理的证明框架,该框架由3个原创性概念和1个定理构成,其中3个概念是空间信息、熵误差和抽样后验概率估计。参数估计定理证明,熵误差是可达的,反之,任何估计器的熵误差不低于熵误差。空间信息论的建立将对雷达探测的系统理论和方法产生巨大的推动作用。     

Entropy invariance for autoregressive processes constructed by linear filtering

In signal processing and in computational techniques for applied mathematics, linear filtering is an important way of turning a time series with independent innovations into a process with highly dependent signals. Stationary autoregressive processes are some of the best-studied linear modes, and their construction involves either full or soft (randomized) linear filters. The entropy rate (per unit time entropy) of a stationary time series is an important quantitative characteristic. When filtering, the entropy for n-dimensional blocks should change as ‘the order’ increases, but the entropy rate of the process could remain invariant. This is the main issue addressed in the paper, where we prove that full or soft linear filtering for Gaussian or uniform innovations produce no change in the entropy rate of the process. That is, linear operators applied to independent innovations do not add or wipe entropy from the innovations family. The plug-in estimators of the entropy rates are also provided, and they can be used to characterize Gaussian or uniform stationary sources. A simulation study is conducted in order to validate the theoretical results and to give a statistical characterization of the plug-in estimator we propose for the entropy rate. Not only it is easy to calculate and has a good precision, but it bridges the lack for nonparametric estimators for the entropy rate of stationary AR processes.     

Entropy-based kernel mixture modeling for topographic map formation

A new information-theoretic learning algorithm for kernel-based topographic map formation is introduced. In the one-dimensional case, the algorithm is aimed at uniformizing the cumulative distribution of the kernel mixture densities by maximizing its differential entropy. A nonparametric differential entropy estimator is used on which normalized gradient ascent is performed. Both differentiable and nondifferentiable kernels are in principle supported, such as Gaussian and rectangular (on/off) kernels. The relation is shown with joint entropy maximization of the kernel outputs. The learning algorithm's performance is assessed and compared with the theoretically optimal performance. A fixed-point rule is derived for the case of heterogeneous kernel mixtures. Finally, an extension of the algorithm to the multidimensional case is suggested.     

QUADRATIC TSALLIS ENTROPY BIAS AND GENERALIZED MAXIMUM ENTROPY MODELS

In density estimation task, Maximum Entropy (Maxent) model can effectively use reliable prior information via nonparametric constraints, that is, linear constraints without empirical parameters. However, reliable prior information is often insufficient, and parametric constraints becomes necessary but poses considerable implementation complexity. Improper setting of parametric constraints can result in overfitting or underfitting. To alleviate this problem, a generalization of Maxent, under Tsallis entropy framework, is proposed. The proposed method introduces a convex quadratic constraint for the correction of (expected) quadratic Tsallis Entropy Bias (TEB). Specifically, we demonstrate that the expected quadratic Tsallis entropy of sampling distributions is smaller than that of the underlying real distribution with regard to frequentist, Bayesian prior, and Bayesian posterior framework, respectively. This expected entropy reduction is exactly the (expected) TEB, which can be expressed by the closed‐form formula and acts as a consistent and unbiased correction with an appropriate convergence rate. TEB indicates that the entropy of a specific sampling distribution should be increased accordingly. This entails a quantitative reinterpretation of the Maxent principle. By compensating TEB and meanwhile forcing the resulting distribution to be close to the sampling distribution, our generalized quadratic Tsallis Entropy Bias Compensation (TEBC) Maxent can be expected to alleviate the overfitting and underfitting. We also present a connection between TEB and Lidstone estimator. As a result, TEB–Lidstone estimator is developed by analytically identifying the rate of probability correction in Lidstone. Extensive empirical evaluation shows promising performance of both TEBC Maxent and TEB‐Lidstone in comparison with various state‐of‐the‐art density estimation methods.     

Urban Density and Entropy Functions

Abstract

This paper presents an approach to geographical hypothesis testing based on the concept of expected information. The expected information formula and its relationship to other entropy formulas is first introduced and this concept is then used to test various hypotheses concerning the distribution of population and its density, in the New York, London, and Los Angeles regions. A related method of analysis based on the idea of deriving equivalent forms of system in which entropy is maximized and expected information minimized, is then presented and this provides alternative ways in which the various hypotheses can be tested. Finally, the use of the spatial entropy formula in fitting continuous population density functions to cities is explored and some comparative tests with other methods of estimation are presented. The ease with which the entropy estimator method can be used in this manner is then offset against its disadvantages, and in conclusion, these techniques are drawn together and evaluated.     

用于非高斯系统降维的最小残差熵主元网络

基于最小均方误差的主元分析和主元神经网络是有效的多变量降维统计技术,它们所提取的主元含有系统最大方差.非高斯随机系统的近似模型应当含有系统最大信息熵,但包含最大方差并不一定包含最大信息熵.该文提出一种以最小残差熵为通用指标的非线性主元神经网络模型,并给出了一种基于Parzen窗口密度函数估计的熵近似计算方法和网络学习算法.然后从信息论角度分析了,在高斯随机系统中基于最小残差熵和最小均方差为指标的主元网络学习结果具有一致性.最后以仿真验证该方法的有效性,并与基于最小均方误差的主元分析和主元神经网络方法的计算结果进行对比性分析.