Conditional probability density function estimation with sigmoidalneural networks

Previous developments in conditional density estimation have used neural nets to estimate statistics of the distribution or the marginal or joint distributions of the input-output variables. We modify the joint distribution estimating sigmoidal neural network to estimate the conditional distribution. Thus, the probability density of the output conditioned on the inputs is estimated using a neural network. We derive and implement the learning laws to train the network. We show that this network has computational advantages over a brute force ratio of joint and marginal distributions. We also compare its performance to a kernel conditional density estimator in a larger scale (higher dimensional) problem simulating more realistic conditions.     

Symmetrical overhead in flow networks

Recent work by one of the authors has identified the average mutual information and the conditional entropy as two measures from information theory that are useful in quantifying the system organization and incoherence, respectively. While the scaled average mutual information, or network ascendency, is inherently symmetrical with respect to inputs and outputs, the scaled conditional entropy, or overhead, remains asymmetrical. Employing the joint entropy, instead of the conditional entropy, to characterize the overhead, results in a symmetrical overhead and also permits the decomposition of the system capacity, or complexity, into components useful in following the response of the whole system to perturbations.     

图像配准中图像互信息的快速估计算法

快速、精确地估计图像互信息是图像配准中一个非常重要的课题,它涉及到两幅图像的联合概率密度和边缘概率密度的估计。针对核密度估计法运算量大而导致互信息估计速度慢的问题,提出了一种快速核密度估计法,并用它估计图像互信息。快速算法利用了单位冲激函数性质和基于快速傅立叶变换的快速卷积算法,能在线性时间复杂度内估计互信息。采用临床MRI图像的实验证实了快速算法的性能。     

Executives’ perceived environmental uncertainty shortly after 9/11

Environmental uncertainty refers to situations when decision makers experience difficulty in predicting their organizations’ environments. Prediction difficulty is mapped by closeness of decision makers’ probability distributions of environmental variables to the uniform distribution. A few months after the 9/11 terrorist attacks, we solicited probabilities for three environmental variables from 93 business executives by a mail survey. Each executive assigned probabilities to the future state of the economy specified as categories of growth projected for a year after the 9/11 jolt, conditional probabilities of its effect on her/his organization, and conditional probabilities of her/his organizational response capability to each economic condition. Shannon entropy maps uncertainty, but the data do not provide trivariate state-effect-response distribution. We use maximum entropy method to impute the trivariate distributions from the data on state-effect and state-response bivariate probabilities. Uncertainty about each executive’s probability distribution is taken into account in two ways: using a Dirichlet model with each executive’s distribution as its mode, and using a Bayesian hierarchical model for the entropy. Both models reduce the observed heterogeneity among the executives’ environmental uncertainty. A Bayesian regression examines the effects of two organizational characteristics on uncertainty. Presentation of results includes uncertainty tableaux for visualizations of the joint and marginal entropies and mutual information between variables.     

利用归一化互信息进行基于像斑的遥感影像变化检测

提出了利用遥感影像分割获取像斑进行变化检测的方法。将归一化互信息引入到遥感影像变化检测中,利用像斑的灰度直方图以及联合灰度直方图计算像斑的归一化互信息,依据条件熵最小的原则,获取最佳划分阈值,并与相关系数法进行了比较。实验结果显示归一化互信息法更适用于遥感影像变化检测。     

Model-based decoding, information estimation, and change-point detection techniques for multineuron spike trains

One of the central problems in systems neuroscience is to understand how neural spike trains convey sensory information. Decoding methods, which provide an explicit means for reading out the information contained in neural spike responses, offer a powerful set of tools for studying the neural coding problem. Here we develop several decoding methods based on point-process neural encoding models, or forward models that predict spike responses to stimuli. These models have concave log-likelihood functions, which allow efficient maximum-likelihood model fitting and stimulus decoding. We present several applications of the encoding model framework to the problem of decoding stimulus information from population spike responses: (1) a tractable algorithm for computing the maximum a posteriori (MAP) estimate of the stimulus, the most probable stimulus to have generated an observed single- or multiple-neuron spike train response, given some prior distribution over the stimulus; (2) a gaussian approximation to the posterior stimulus distribution that can be used to quantify the fidelity with which various stimulus features are encoded; (3) an efficient method for estimating the mutual information between the stimulus and the spike trains emitted by a neural population; and (4) a framework for the detection of change-point times (the time at which the stimulus undergoes a change in mean or variance) by marginalizing over the posterior stimulus distribution. We provide several examples illustrating the performance of these estimators with simulated and real neural data.     

Efficient Markov chain Monte Carlo methods for decoding neural spike trains

Stimulus reconstruction or decoding methods provide an important tool for understanding how sensory and motor information is represented in neural activity. We discuss Bayesian decoding methods based on an encoding generalized linear model (GLM) that accurately describes how stimuli are transformed into the spike trains of a group of neurons. The form of the GLM likelihood ensures that the posterior distribution over the stimuli that caused an observed set of spike trains is log concave so long as the prior is. This allows the maximum a posteriori (MAP) stimulus estimate to be obtained using efficient optimization algorithms. Unfortunately, the MAP estimate can have a relatively large average error when the posterior is highly nongaussian. Here we compare several Markov chain Monte Carlo (MCMC) algorithms that allow for the calculation of general Bayesian estimators involving posterior expectations (conditional on model parameters). An efficient version of the hybrid Monte Carlo (HMC) algorithm was significantly superior to other MCMC methods for gaussian priors. When the prior distribution has sharp edges and corners, on the other hand, the "hit-and-run" algorithm performed better than other MCMC methods. Using these algorithms, we show that for this latter class of priors, the posterior mean estimate can have a considerably lower average error than MAP, whereas for gaussian priors, the two estimators have roughly equal efficiency. We also address the application of MCMC methods for extracting nonmarginal properties of the posterior distribution. For example, by using MCMC to calculate the mutual information between the stimulus and response, we verify the validity of a computationally efficient Laplace approximation to this quantity for gaussian priors in a wide range of model parameters; this makes direct model-based computation of the mutual information tractable even in the case of large observed neural populations, where methods based on binning the spike train fail. Finally, we consider the effect of uncertainty in the GLM parameters on the posterior estimators.     

On computation of information via variation and inequalities for the entropy function

This paper supplements the author’s paper [1]. We obtain an explicit formula which in a special case allows us to calculate the maximum of mutual information of several random variables via the variational distance between the joint distribution of these random variables and the product of their marginal distributions. We establish two new inequalities for the binary entropy function, which are related to the problem considered here.     

互信息的序决策信息系统属性约简研究

优势关系粗糙集理论是粗糙集理论有意义的推广,决策信息系统知识约简是粗糙集理论的核心内容之一.通过在协调序决策信息系统中引入条件熵、互信息概念,给出了基于条件熵、互信息的协调序决策信息系统属性约简算法,并通过学生评价决策信息系统验证了该算法的有效性,使协调序决策信息系统的属性约简得到了扩展.在不协调序决策信息系统中引入限定条件熵、限定互信息概念,并给出基于限定互信息的不协调序决策信息系统属性约简算法,为不协调序决策信息系统的属性约简的应用提供了可行的解决方法.     

Mutual information of several random variables and its estimation via variation

We obtain some upper and lower bounds for the maximum of mutual information of several random variables via variational distance between the joint distribution of these random variables and the product of its marginal distributions. In this connection, some properties of variational distance between probability distributions of this type are derived. We show that in some special cases estimates of the maximum of mutual information obtained here are optimal or asymptotically optimal. Some results of this paper generalize the corresponding results of [1–3] to the multivariate case.     

Bayes网络最优近似下边缘分布的不变性

删除Bayes网络中的弧以减小网络结构的复杂性，从而降低概率推理算法的复杂度是一种对Bayes网络进行近似的方法．该文讨论了在删除Bayes网络中的一条弧之后得到的最优近似概率分布和原概率分布之间的关系，证明了对满足一定条件的结点子集而言，其边缘概率分布在近似以后具有不变性．     

基于归一化互信息向量熵的多幅图像配准方法

提出了一种新的多幅图像配准方法,归一化互信息向量熵方法。这种方法先计算任意两幅图像间的联合概率分布,然后根据联合概率分布计算它们间的归一化互信息,把所有两幅图像组合得到的归一化互信息组成一个向量,最后计算该归一化互信息向量的熵。最大熵对应最佳配准位置。通过对人体脑部图像的刚体配准实验,从函数曲线、计算时间和配准精度方面,对新方法和其它三种方法进行了分析和比较。实验结果表明,新提出的方法可以提高配准精度、减少配准时间。     

基于信息熵的形式背景属性约简

属性重要度和属性约简都是形式概念分析研究中的关注重点.通过信息粒的角度,文中提出基于信息熵研究形式背景的属性约简的一些方法.首先,给出形式背景的信息熵、条件熵及互信息等定义,通过条件熵对协调的决策形式背景进行属性约简,得到形式背景的粒协调和熵协调是等价的.然后,在熵不协调的决策形式背景中定义有限信息熵、有限条件熵和有限互信息,利用有限条件熵对不协调的决策形式背景进行属性约简.最后,基于属性重要度分别设计熵协调和熵不协调的决策形式背景的属性约简算法,通过数值实验验证文中算法的有效性.     

Information and Coding Discrimination of Pseudo-Additive Entropies (PAE)

PAE cannot be made a basis for either a generalized statistical mechanics or a generalized information theory. Either statistical independence must be waived, or the expression of the averaged conditional probability as the difference between the marginal and joint entropies must be relinquished. The same inequality, relating the PAE to the Rényi entropy, when applied to the mean code length produces an expression that is without bound as the order of the code length approaches infinity. Since the mean code length associated with the Rényi entropy is finite and can be made to come as close to the Hartley entropy as desired in the same limit, the PAE have a more limited range of validity than the Rényi entropy which they approximate.     

Probabilistic distances between finite-state finite-alphabet hidden Markov models

This note considers the problem of evaluating a probabilistic distance between discrete-time, homogeneous, first-order, finite-state finite-alphabet hidden Markov models (HMMs). Our approach is based on a correspondence between probability measures and HMMs established in this note. Using a probability measure transformation technique, we obtain recursive expressions for the relative entropy between the marginal probability distributions of two HMMs under consideration. Also, the relative entropy rate, as the limit of the time-averaged value of the above relative entropy, is obtained. These expressions are given in terms of the parameters of the given HMMs. Furthermore, we show that the probabilistic distance between HMMs used in the existing literature admits a representation in terms of a conditional expectation given the observation sequence. This representation allows us to evaluate this distance using an information state approach.     

Minimax mutual information approach for independent component analysis

Minimum output mutual information is regarded as a natural criterion for independent component analysis (ICA) and is used as the performance measure in many ICA algorithms. Two common approaches in information-theoretic ICA algorithms are minimum mutual information and maximum output entropy approaches. In the former approach, we substitute some form of probability density function (pdf) estimate into the mutual information expression, and in the latter we incorporate the source pdf assumption in the algorithm through the use of nonlinearities matched to the corresponding cumulative density functions (cdf). Alternative solutions to ICA use higher-order cumulant-based optimization criteria, which are related to either one of these approaches through truncated series approximations for densities. In this article, we propose a new ICA algorithm motivated by the maximum entropy principle (for estimating signal distributions). The optimality criterion is the minimum output mutual information, where the estimated pdfs are from the exponential family and are approximate solutions to a constrained entropy maximization problem. This approach yields an upper bound for the actual mutual information of the output signals - hence, the name minimax mutual information ICA algorithm. In addition, we demonstrate that for a specific selection of the constraint functions in the maximum entropy density estimation procedure, the algorithm relates strongly to ICA methods using higher-order cumulants.     

两阶段领域自适应学习

针对领域自适应问题中源域和目标域的联合分布差异最小化问题,提出两阶段领域自适应学习方法.在第一阶段考虑样本标签和数据结构的判别信息,通过学习一个共享投影变换,使投影后的共享空间中边缘分布的差异最小.第二阶段利用源域标记数据和目标域非标记数据学习一个带结构风险的自适应分类器,不仅能最小化源域和目标域条件分布差异,还能进一步保持源域和目标域边缘分布的流形一致性.在3个基准数据集上的实验表明,文中方法在平均分类准确率和Kappa系数两项评价指标上均表现较优.     

Visual domain adaptation via transfer feature learning

One of the serious challenges in computer vision and image classification is learning an accurate classifier for a new unlabeled image dataset, considering that there is no available labeled training data. Transfer learning and domain adaptation are two outstanding solutions that tackle this challenge by employing available datasets, even with significant difference in distribution and properties, and transfer the knowledge from a related domain to the target domain. The main difference between these two solutions is their primary assumption about change in marginal and conditional distributions where transfer learning emphasizes on problems with same marginal distribution and different conditional distribution, and domain adaptation deals with opposite conditions. Most prior works have exploited these two learning strategies separately for domain shift problem where training and test sets are drawn from different distributions. In this paper, we exploit joint transfer learning and domain adaptation to cope with domain shift problem in which the distribution difference is significantly large, particularly vision datasets. We therefore put forward a novel transfer learning and domain adaptation approach, referred to as visual domain adaptation (VDA). Specifically, VDA reduces the joint marginal and conditional distributions across domains in an unsupervised manner where no label is available in test set. Moreover, VDA constructs condensed domain invariant clusters in the embedding representation to separate various classes alongside the domain transfer. In this work, we employ pseudo target labels refinement to iteratively converge to final solution. Employing an iterative procedure along with a novel optimization problem creates a robust and effective representation for adaptation across domains. Extensive experiments on 16 real vision datasets with different difficulties verify that VDA can significantly outperform state-of-the-art methods in image classification problem.     

A non-reference image fusion metric based on mutual information of image features

The widespread usage of image fusion causes an increase in the importance of assessing the performance of different fusion algorithms. The problem of introducing a suitable quality measure for image fusion lies in the difficulty of defining an ideal fused image. In this paper, we propose a non-reference objective image fusion metric based on mutual information which calculates the amount of information conducted from the source images to the fused image. The considered information is represented by image features like gradients or edges, which are often in the form of two-dimensional signals. In this paper, a method of estimating the joint probability distribution from marginal distributions is also presented which is employed in calculation of mutual information. The proposed method is compared with the most popular existing algorithms. Various experiments, performed on several databases, certify the efficiency of our proposed method which is more consistent with the subjective criteria.     

基于最小联合互信息亏损的最优特征选择算法

提出了一种基于最小联合互信息亏损的最优特征选择算法。该算法首先通过一种动态渐增策略搜索一个特征全集的无差异特征子集,并基于最小条件互信息原则在保证每一步中联合互信息量亏损都最小的情况下筛选其中的冗余特征,从而得到一个近似最优特征子集。针对现有基于条件互信息的条件独立性测试方法在高维特征域上所面临的效率瓶颈问题,给出了一种用于估计条件互信息的快速实现方法,并将其用于所提算法的实现。分类实验结果表明,所提算法优于经典的特征选择算法。此外,执行效率实验结果表明,所提条件互信息的快速实现方法在执行效率上有着显著的优势。