Properties of cross-entropy minimization

The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values. Various fundamental properties of cross-entropy minimization are proven and collected in one place. Cross-entropy's well-known properties as an information measure are extended and strengthened when one of the densities involved is the result of cross-entropy minimization. The interplay between properties of cross-entropy minimization as an inference procedure and properties of cross-entropy as an information measure is pointed out. Examples are included and general analytic and computational methods of finding minimum cross-entropy probability densities are discussed.     

Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy

Jaynes's principle of maximum entropy and Kullbacks principle of minimum cross-entropy (minimum directed divergence) are shown to be uniquely correct methods for inductive inference when new information is given in the form of expected values. Previous justifications use intuitive arguments and rely on the properties of entropy and cross-entropy as information measures. The approach here assumes that reasonable methods of inductive inference should lead to consistent results when there are different ways of taking the same information into account (for example, in different coordinate system). This requirement is formalized as four consistency axioms. These are stated in terms of an abstract information operator and make no reference to information measures. It is proved that the principle of maximum entropy is correct in the following sense: maximizing any function but entropy will lead to inconsistency unless that function and entropy have identical maxima. In other words given information in the form of constraints on expected values, there is only one (distribution satisfying the constraints that can be chosen by a procedure that satisfies the consistency axioms; this unique distribution can be obtained by maximizing entropy. This result is established both directly and as a special case (uniform priors) of an analogous result for the principle of minimum cross-entropy. Results are obtained both for continuous probability densities and for discrete distributions.     

The SURE approach to reliability analysis

The SURE computer program, a reliability-analysis tool for ultrareliable computer-system architectures, provides an efficient means for computing reasonably accurate upper and lower bounds for the death state probabilities of a large class of semi-Markov models. Once a semi-Markov model is described using a simple input language, SURE automatically computes the upper and lower bounds on the probability of system failure. A parameter of the model can be specified as a variable over a range of values, thus directing SURE to perform a sensitivity analysis automatically. The program provides a rapid computational capability for semi-Markov models useful for describing the fault-handling behavior of fault-tolerant computer systems. The only modeling restriction imposed by the program is that the nonexponential recovery transitions must be fast in comparison to the mission time. The SURE reliability-analysis method uses a fast bounding theorem based on means and variances and yields upper and lower bounds on the probability of system failure. Techniques have been developed to enable SURE to solve models with loops and calculate the operational-state probabilities. The computation is extremely fast, and large state-spaces can be directly solved; a pruning technique enables SURE to process extremely large models     

Recursive solution technique in a multi-server bi-level queueingsystem with server breakdowns

The authors present a multiserver, first-come first-served queuing system that alternates between two modes of system operation. In one mode, all s servers are available, and in the other mode, only s-1 servers are available for serving the customers. This is due to breakdown of one of the servers. The random variables representing the system with s servers and s-1 servers have exponential distributions. In such a system, the steady-state birth/death equations are coupled because of the two modes of operation. A recursive solution is presented for computing the steady-state probabilities of such a system. Once these probabilities are known, the performance measures of interest can be easily obtained. Two practical examples validate the results and show the utility of this method. A distinct advantage of the recursive technique is that it is much faster and requires much less memory than the existing nonrecursive techniques. In a bilevel situation, the system performance measures are always bounded by two independent queuing systems with s and s-1 servers. A procedure has been outlined for extension to multiple modes of system operation     

Integrated physics-oriented statistical modeling, simulation, andoptimization [MESFETs]

Physics-based modeling of MESFETs is addressed from the point of view of efficient simulation, accurate behavior prediction and robust parameter extraction. A novel integration of a large-signal physics-based model into the harmonic balance equations for simulation of nonlinear circuits, involving an efficient Newton update, is presented and exploited in a gradient-based FAST (feasible adjoint sensitivity technique) circuit optimization technique. For yield-driven MMIC design a relevant physics-based statistical modeling methodology is presented. Quadratic approximation of responses and gradients suitable for yield optimization is discussed. The authors verify their theoretical contributions and exemplify their computational results using built-in and user-programmable modeling capabilities of the CAE systems OSA90/hope and HarPE. Results of device modeling using a field-theoretic nonlinear device simulator are reported     

Truncated sequential hypothesis tests

Through a careful examination of the equations by which Wald determines the values of the boundaries for tests of sequential hypotheses, we are able to obtain interesting relationships for the conditional probability distributions of the stage at which the test terminates. These enable us to study sequential tests in which the boundaries are functions of the sample number. We are particularly concerned with tests with convergent boundaries, and investigate a set of boundaries which approach those of a truncated Wald test. Approximate expressions for the expected sample number and probabilities of error are obtained for the tests considered. The obtained approximations apply best for gently tapering slopes. Extensions of our method can be applied to evaluate various ad hoc schemes for truncating tests and to the theory of tests with a varying parameter.     

A bound on the rate-distortion function and application to images

An upper bound on the rate-distortion function for discrete ergodic sources with memory is developed by partitioning the source sample space into a finite number of disjoint subsets and bounding the rates for each subset. The bound depends only on the mean vectors and covariance matrices for the subsets and is easy to compute. It is tighter than the Gaussian bound for sources that exhibit clustering of either the values or covariances of successive source outputs. The bound is evaluated for a certain class of pictorial data using both one-dimensional and two-dimensional blocks of picture elements. Two-dimensional blocks yield a tighter bound than one-dimensional blocks; both result in a significantly tighter bound than the Gaussian bound.     

Multilevel codes for unequal error protection

Two combined unequal error protection (UEP) coding and modulation schemes are proposed. The first method multiplexes different coded signal constellations, with each coded constellation providing a different level of error protection. In this method, a codeword specifies the multiplexing rule and the choice of the codeword from a fixed codebook is used to convey additional important information. The decoder determines the multiplexing rule before decoding the rest of the data. The second method is based on partitioning a signal constellation into disjoint subsets in which the most important data sequence is encoded, using most of the available redundancy, to specify a sequence of subsets. The partitioning and code construction is done to maximize the minimum Euclidean distance between two different valid subset sequences. This leads to ways of partitioning the signal constellations into subsets. The less important data selects a sequence of signal points to be transmitted from the subsets. A side benefit of the proposed set partitioning procedure is a reduction in the number of nearest neighbors, sometimes even over the uncoded signal constellation     

Joint spectrum load balancing and handoff management in cognitive radio networks: a non-cooperative game approach

We propose a non-cooperative game theory based algorithm for spectrum management problem in cognitive radio networks taking into account the spectrum handoff effects. The objective is to minimize the spectrum access time of Secondary Users (SUs) which are competing for spectrum opportunities in heterogeneous environment. In this paper, the preemptive resume priority (PRP) M/G/1 queuing model is used to characterize the multiple handoff and data delivery time of SUs. Also an explicit solution for channels selection probabilities of each SU is extracted for PRP M/M/1 model specifically. The effect of handoffs is considered as the interrupted packets which return to the SUs’ low priority queue when the high priority Primary User’s packets are arrived to take service. The queuing delay of SUs’ and the effect of these returned packets are considered in order to balance the load of SUs on channels so that the minimum spectrum access time is sensed by each SU. The non-cooperative spectrum load balancing with handoff management game is proposed to find a distributed solution for each SU. It is shown that this game has a unique Nash equilibrium point which can be achieved by SUs as decision makers. At this equilibrium, each SU incurs the minimum delay on all channels while the free spectrum holes of channels are utilized efficiently. Simulation results are provided to evaluate the performance of the proposed scheme in terms of spectrum access delay, fairness, and channels spectrum holes utilization.     

基于事件触发的通信业务量仿真方法

经典的排队论分析方法一般通过建立Markov链模型,对其稳态参数进行求解,但在Markov链状态数较多时,该求解平衡方程的过程将变得较为困难,甚至于无法求解。采用基于事件调度的离散系统仿真方法,给出了一种排队系统的仿真方法,并对一个排队系统进行了蒙特卡罗(MonteCarlo)仿真。仿真结果证明,该方法求解迅速,结果误差较小。     

基于网格的最小交叉熵算法在PET图像重建中的应用

基于内容的自适应三角形网格模型是描述图像的一种有效方法,本文将网格模型与最小交叉熵算法相结合,并加入先验解剖信息,用于PET图像重建.在本文提出的新算法中,先将投影数据用滤波反投影方法(FBP)生成参考图像,再对参考图像提取网格节点,用加入先验解剖信息的最小交叉熵算法对网格节点灰度值进行迭代计算,最后利用迭代后的网格节点灰度值对象素点进行插值得到重建后的图像.在仿真实验中,将该算法与最大似然方法(MLEM)等算法作比较,并分析了参数对重建结果的影响.     

Fault-tolerant programs and their reliability

The authors review and extend available techniques for achieving fault-tolerant programs. The representation of the techniques is uniform and is illustrated by simple examples. For each technique a fault tree has been developed to derive failure probability from the probabilities of the basic fault events. This allows the subsequent analysis of program-failure causes and the reliability modeling of computer programs. Numerical examples are given to support the comparison of the reviewed techniques. The models can be used to evaluate numerical values of program reliability in a relatively simple way. The models deal with program reliability for a single run, which seems more practical and straightforward than dealing with distributions as for hardware systems. Evaluations obtained by using models correspond to those used in the literature; however, the authors' procedures are computationally simpler     

Hybrid analysis of response time distributions in queueing networks

A hybrid analytic/simulation methodology is formulated for evaluating end-to-end response time distributions in closed product-form queueing networks. The method combines Markov-Monte-Carlo simulation with analytical results pertaining to uniformized Markov chains and product-form queuing networks. A stratified sampling plan is incorporated as a variance reduction technique. The concept of importance sampling is used to reduce the computational requirements of the plan and make it realizable in practice. The most important consequence of applying uniformization is that it enables the characterization of the response time distribution as an infinite mixture of Erlangian distributions. An estimate of the entire response time distribution may then be obtained in each simulation trial. This circumvents the practical problems associated with estimating tail probabilities. A numerical example is provided to illustrate the theory     

Stochastic behavior of a two-dissimilar-unit standby redundant system with repair maintenance

A standby redundant system of two dissimilar units is considered under the assumptions that both the failure and the repair time distributions are arbitrary. The system states are defined corresponding to the failure of the specified units. The first passage time distributions, the expected numbers of visits to a certain state, the transition probabilities, and the limiting probabilities are obtained using the unique modifications of the regeneration point techniques in Markov renewal processes. Two particular cases are finally presented.     

Blocking Probabilities for a Class of Spiderweb Channel Graphs

The blocking probabilities for spiderweb channel graphs are well known to be difficult to compute. Recently, Takagi studied a class of spiderweb channel graphs and gave recursive equations for computing their blocking probabilities. In this paper we use a recent result in combinatorics to give a dosed form solution for the recursive equations. Consequently, the blocking probability for any graph in this class can be expressed in a form involving at most two nested summations.     

Covering properties of convolutional codes and associated lattices

The paper describes Markov methods for analyzing the expected and worst case performance of sequence-based methods of quantization. We suppose that the quantization algorithm is dynamic programming, where the current step depends on a vector of path metrics, which we call a metric function. Our principal objective is a concise representation of these metric functions and the possible trajectories of the dynamic programming algorithm. We shall consider quantization of equiprobable binary data using a convolutional code. Here the additive group of the code splits the set of metric functions into a finite collection of subsets. The subsets form the vertices of a directed graph, where edges are labeled by aggregate incremental increases in mean squared error (MSE). Paths in this graph correspond both to trajectories of the Viterbi algorithm and to cosets of the code. For the rate 1/2 convolutional code [1+D², 1+D+D²], this graph has only nine vertices. In this case it is particularly simple to calculate per dimension expected and worst case MSE, and performance is slightly better than the binary [24, 12] Golay code. Our methods also apply to quantization of arbitrary symmetric probability distributions on [0, 1] using convolutional codes. For the uniform distribution on [0, 1], the expected MSE is the second moment of the “Voronoi region” of an infinite-dimensional lattice determined by the convolutional code. It may also be interpreted as an increase in the reliability of a transmission scheme obtained by nonequiprobable signaling. For certain convolutional codes we obtain a formula for expected MSE that depends only on the distribution of differences for a single pair of path metrics     

Approximate steady-state solutions for Markov systems based upon the transition-rate matrix deviation concept

This paper derives an approximate method for solving the steady-state Markov equations. The approximate solutions are determined using the known or easily obtainable solutions of a Markov system whose transition-rate matrix slightly differs from that of the original system. The method suggested makes it possible to derive approximate analytic expressions for the steady-state probabilities of system states and associated reliability indices and/or to calculate their numerical values. The expressions for the error bounds for the method are provided. The method is applied to derive expressions for the state probabilities of two transmission lines sharing the same right of way and exposed to common cause failures, as a demonstration.     

On the computation of the performance probabilities for block codeswith a bounded-distance decoding rule

When a block code is used on a discrete memoryless channel with an incomplete decoding rule that is based on a generalized distance, the probability of decoding failure, the probability of erroneous decoding, and the expected number of symbol decoding errors can be expressed in terms of the generalized weight enumerator polynomials of the code. For the symmetric erasure channel, numerically stable methods to compute these probabilities or expectations are proposed for binary codes whose distance distributions are known, and for linear maximum distance separable (MDS) codes. The method for linear MDS codes saves the computation of the weight distribution and yields upper bounds for the probability of erroneous decoding and for the symbol error rate by the cumulative binomial distribution. Numerical examples include a triple-error-correcting Bose-Chaudhuri-Hocquenghem (BCH) code of length 63 and a Reed-Solomon code of length 1023 and minimum distance 31     

Some inequalities concerning random subsets of a set (Corresp.)

A class of plausible inequalities between conditional probabilities is considered involving random subsets of a set which arise, for example, in studying the capacity of a multiple access channel. Some of the inequalities are true and can be proved by the Fortuin-Kasteleyn-Ginibre (FKG) inequality. Some others are true but seem to need other methods. Still others are false. The results also apply to studying the conditional probability of the failure of a device designed to achieve reliability by using redundant parts, given side information about the failure of some of its components.