Universal variable-to-fixed length source codes

A universal variable-to-fixed length algorithm for binary memoryless sources which converges to the entropy of the source at the optimal rate is known. We study the problem of universal variable-to-fixed length coding for the class of Markov sources with finite alphabets. We give an upper bound on the performance of the code for large dictionary sizes and show that the code is optimal in the sense that no codes exist that have better asymptotic performance. The optimal redundancy is shown to be H log log M/log M where H is the entropy rate of the source and M is the code size. This result is analogous to Rissanen's (1984) result for fixed-to-variable length codes. We investigate the performance of a variable-to-fixed coding method which does not need to store the dictionaries, either at the coder or the decoder. We also consider the performance of both these source codes on individual sequences. For individual sequences we bound the performance in terms of the best code length achievable by a class of coders. All the codes that we consider are prefix-free and complete     

A transform tree code for stationary Gaussian sources

A new tree code is introduced for discrete-time stationary Gaussian sources with hounded, integrable power spectra and the squared-error distortion measure. The codewords in the tree are reconstructions of Karhunen-Loève transforms of the source words. The branching factor and the number of code letters per branch may vary with level in the tree. A theorem that guarantees the existence of an optimal code for any code rate using such a tree is proved. The proof uses the random coding argument in conjunction with a theorem on survival of a branching process with random environment. A suboptimal but computationally affordable realization of the theorem's coding technique was used for encoding simulations for six autoregressive sources at rates of1.0, 0.50, 0.25, and0.10bits per source symbol. The average distortion results were generally within1dB of the distortion-rate bound but varied widely depending on the source and rate. The results were compared with those for transform quantization simulations for the same sources and rates. The tree code always performed better but only by an average of0.44dB all sources and rates. Longer source blocks and more intensive search would certainly improve the performance of the tree codes, but at the expense of extra computation and storage.     

Second-Order Asymptotics in Fixed-Length Source Coding and Intrinsic Randomness

There is a difference between the optimal rates of fixed-length source coding and intrinsic randomness when we care about the second-order asymptotics. We prove this difference for general information sources and then investigate independent and identically distributed (i.i.d.) random variables and Markovian variables as examples. The difference is demonstrated through an investigation of the second-order asymptotic behavior of the rates. A universal fixed-length source code attaining the second-order optimal rate is also proposed. The difference between the rates of fixed-length source coding and intrinsic randomness proves that the outputs of fixed-length source codes are not uniformly distributed.      

On the average codeword length of optimal binary codes for extended sources (Corresp.)

Although optimal binary source coding using symbol blocks of increasing length must eventually yield a code having an average codeword length arbitrarily close to the source entropy, it is known that the sequence of average codeword lengths need not be nonincreasing. The sequence is, however, bounded above by the average codeword length of the source, and certain subsequences must be nondecreasing. Sufficient conditions are obtained describing sources for which a decrease in average codeword length is achieved when coding pairs of symbols. A sufficient condition specifying sources for which no such decrease is possible is also obtained.     

基于分离信源信道码的相关信源在有噪广播信道下的可靠和安全传输

从信息论的角度对相关信源在离散无记忆广播信道下可靠和安全传输的问题进行研究。2个信源经过有噪信道分别到达各自指定的目的节点并被无损恢复,同时还要保证信源信息对于非指定的目的节点要有一定的保密性。采用信源信道分离的随机码策略,得到相关信源在一般广播信道下能够可靠和安全传输的充分条件。当2个信源的公共信息为二者的互信息时,可获得最佳压缩传输效率,并且能够做到信源信息传输的部分绝对保密。当广播信道采用退化信源集或满足more capable广播信道性质时,得到了可靠和安全传输的充分必要条件,此时分离信源信道码为最优码。     

Universal source coding of finite alphabet sources via composition classes

The universal source coding theorem for stationary sources states that, by using a preselected code with sufficiently large block length, any stationary source can be encoded arbitrarily close to the optimum performance theoretically attainable for the source. The code construction typically used to prove this theorem offers no performance guarantees for nonstationary sources. A code construction (based on composition classes) is described which is shown to be universal for stationary sources. In addition, this code construction guarantees that the distortion for any block in the sequence will not exceed a maximum value determined by its composition.     

Tree coding of image subbands

The authors consider the encoding of image subbands with a tree code that is asymptotically optimal for Gaussian sources and the mean squared error (MSE) distortion measure. They first prove that optimal encoding of ideally filtered subbands of a Gaussian image source achieves the rate distortion bound for the MSE distortion measure. The optimal rate and distortion allocation among the subbands is a by-product of this proof. A bound is derived which shows that subband coding is closer than full-band coding to the rate distortion bound for a finite length sequence. The tree codes are then applied to encode the image subbands, both nonadaptively and adaptively. Since the tree codes are stochastic and the search of the code tree is selective, a relatively few reproduction symbols may have an associated squared error a hundred times larger than the target for the subband. Correcting these symbols through a postcoding procedure improves the signal-to-noise ratio and visual quality significantly, with a marginal increase in total rate.     

Universal codeword sets and representations of the integers

Countable prefix codeword sets are constructed with the universal property that assigning messages in order of decreasing probability to codewords in order of increasing length gives an average code-word length, for any message set with positive entropy, less than a constant times the optimal average codeword length for that source. Some of the sets also have the asymptotically optimal property that the ratio of average codeword length to entropy approaches one uniformly as entropy increases. An application is the construction of a uniformly universal sequence of codes for countable memoryless sources, in which thenth code has a ratio of average codeword length to source rate bounded by a function ofnfor all sources with positive rate; the bound is less than two forn = 0and approaches one asnincreases.     

A source matching approach to finding minimax codes

A source matching approach is presented to the problem of finding minimax cedes for classes of memoryless sources. The approach leads to a channel capacity problem so that Blahut's algorithm can be used to find approximations to the minimax code. Closed form solutions are presented for the class of monotonic sources and for a class of Bernoulli-like sources. For extensions of finite alphabet memoryless sources, a modified Lynch-Davisson code has performance close to that of the minimax code. The exact solution to the source matching problem and the resulting codes are presented for the extensions of binary codes up to blocklength 31.     

Bounding the average length of optimal source codes via majorization theory

We consider the problem of bounding the average length of an optimal (Huffman) source code when only limited knowledge of the source symbol probability distribution is available. For instance, we provide tight upper and lower bounds on the average length of optimal source codes when only the largest or the smallest source symbol probability is known. Our results rely on basic results of majorization theory and on the Schur concavity of the minimum average length of variable-length source codes for discrete memoryless sources. In the way to prove our main result we also give closed formula expressions for the average length of Huffman codes for several classes of probability distributions.     

Quantizers with uniform encoders and channel optimized decoders

Scalar quantizers with uniform encoders and channel optimized decoders are studied for uniform sources and binary symmetric channels. It is shown that the natural binary code (NBC) and folded binary code (FBC) induce point density functions that are uniform on proper subintervals of the source support, whereas the Gray code (GC) does not induce a point density function. The mean-squared errors (MSE) for the NBC, FBC, GC, and for randomly chosen index assignments are calculated and the NBC is shown to be mean-squared optimal among all possible index assignments, for all bit-error rates and all quantizer transmission rates. In contrast, it is shown that almost all index assignments perform poorly and have degenerate codebooks.     

Mismatched DPCM encoding of autoregressive processes

A method for computing the mean squared error distortion of differential pulse code modulation (DPCM) applied to Gaussian autoregressive sources is developed. This extends previous work wherein the code predictor was matched to the source. A two-dimensional version of the projection method for the computation of the stationary distribution of the joint source-state process is developed from which distortion can be readily evaluated. An iterative algorithm is used to optimize the quantizer for a given source and predictor. The results show that the matched predictor is very nearly optimal but not exactly so, and that DPCM is fairly robust to mismatch of the prediction coefficient to the correlation coefficient of a first-order autoregressive source     

An easily implementable universal code for the binary source (Corresp.)

Recently Gallager has given the construction for an optimal source code for geometrically distributed integer alphabets. Gallager's code can be modified to perform as a universal code for the memoryless binary source; this code construction has several important practical advantages over previous universal codes for the binary source.     

On optimal permutation codes

Permutation codes are vector quantizers whose codewords are related by permutations and, in one variant, sign changes. Asymptotically, as the vector dimension grows, optimal Variant I permutation code design is identical to optimal entropy-constrained scalar quantizer (ECSQ) design. However, contradicting intuition and previously published assertions, there are finite block length permutation codes that perform better than the best ones with asymptotically large length; thus, there are Variant I permutation codes whose performances cannot be matched by any ECSQ. Along similar lines, a new asymptotic relation between Variant I and Variant II permutation codes is established but again demonstrated to not necessarily predict the performances of short codes. Simple expressions for permutation code performance are found for memoryless uniform and Laplacian sources. The uniform source yields the aforementioned counterexamples     

Precise minimax redundancy and regret

Recent years have seen a resurgence of interest in redundancy of lossless coding. The redundancy (regret) of universal fixed-to-variable length coding for a class of sources determines by how much the actual code length exceeds the optimal (ideal over the class) code length. In a minimax scenario one finds the best code for the worst source either in the worst case (called also maximal minimax) or on average. We first study the worst case minimax redundancy over a class of stationary ergodic sources and replace Shtarkov's bound by an exact formula. Among others, we prove that a generalized Shannon code minimizes the worst case redundancy, derive asymptotically its redundancy, and establish some general properties. This allows us to obtain precise redundancy for memoryless, Markov, and renewal sources. For example, we present the exact constant of the redundancy for memoryless and Markov sources by showing that the integer nature of coding contributes log(logm/(m-1))/logm+o(1) where m is the size of the alphabet. Then we deal with the average minimax redundancy and regret. Our approach here is orthogonal to most recent research in this area since we aspire to show that asymptotically the average minimax redundancy is equivalent to the worst case minimax redundancy for some classes of sources. After formulating some general bounds relating these two redundancies, we prove our assertion for memoryless and Markov sources. Nevertheless, we provide evidence that maximal redundancy of renewal processes does not have the same leading term as the average minimax redundancy (however, our general results show that maximal and average regrets are asymptotically equivalent).     

Causal source codes

Causal source codes are defined. These include quantizers, delta modulators, differential pulse code modulators, and adaptive versions of these. Several types of causal codes are identified. For memoryless sources it is shown that the optimum performance attainable by causal codes can be achieved by memoryless codes or by time-sharing memoryless codes. This optimal performance can be evaluated straightforwardly.     

Joint design of fixed-rate source codes and multiresolution channelcodes

We propose three new design algorithms for jointly optimizing source and channel codes. Our optimality criterion is to minimize the average end-to-end distortion. For a given channel SNR and transmission rate, our joint source and channel code designs achieve an optimal allocation of bits between the source and channel coders. Our three techniques include a source-optimized channel code, a channel-optimized source code, and an iterative descent technique combining the design strategies of the other two codes. The joint designs use channel-optimized vector quantization (COVQ) for the source code and rate compatible punctured convolutional (RCPC) coding for the channel code. The optimal bit allocation reduces distortion by up to 6 dB over suboptimal allocations and by up to 4 dB relative to standard COVQ for the source data set considered. We find that all three code designs have roughly the same performance when their bit allocations are optimized. This result follows from the fact that at the optimal bit allocation the channel code removes most of the channel errors, in which case the three design techniques are roughly equivalent. We also compare the robustness of the three techniques to channel mismatch. We conclude the paper by relaxing the fixed transmission rate constraint and jointly optimizing the transmission rate, source code, and channel code     

Distributed source coding using short to moderate length rate-compatible LDPC codes: the entire Slepian-Wolf rate region

In this paper, we propose a scheme for distributed source coding of correlated sources using a single systematic LDPC code. In particular, since we are interested in wireless sensor network applications, we consider LDPC codes with short to moderate lengths that achieve every arbitrary coding rate on the Slepian-Wolf rate region. We simplify the distributed source coding problem to the rate-compatible LDPC code design with an unequal error protection property. The decoders communicate to each other to exchange information bits prior to decoding. However, thereafter, each performs the decoding independently. Therefore, errors in one decoder do not affect the other one. The simulation results confirm that the gap from the theoretical limit remains almost the same for different rates on the Slepian-Wolf rate region. First, we consider two correlated sources. We show that our proposed scheme improves the performance of distributed source coding of two sources considerably. This benefit is more stressed for application with short to moderate length sequences. Then, we study distributed source coding of three sources. As a special case, we investigate three sources that are pairwise correlated with the same correlation probability. We show that the gap from the theoretical limit is smaller than that of previous work. We also investigate the distributed source coding of correlated sources when there is no prior knowledge of the correlation parameter at the time of code design. We note that although the proposed distributed source coding is well suited for sensor networks (where sequences with less than 10000 bits are used), the method can be generalized to other distributed source coding applications.     

Weighted universal image compression

We describe a general coding strategy leading to a family of universal image compression systems designed to give good performance in applications where the statistics of the source to be compressed are not available at design time or vary over time or space. The basic approach considered uses a two-stage structure in which the single source code of traditional image compression systems is replaced with a family of codes designed to cover a large class of possible sources. To illustrate this approach, we consider the optimal design and use of two-stage codes containing collections of vector quantizers (weighted universal vector quantization), bit allocations for JPEG-style coding (weighted universal bit allocation), and transform codes (weighted universal transform coding). Further, we demonstrate the benefits to be gained from the inclusion of perceptual distortion measures and optimal parsing. The strategy yields two-stage codes that significantly outperform their single-stage predecessors. On a sequence of medical images, weighted universal vector quantization outperforms entropy coded vector quantization by over 9 dB. On the same data sequence, weighted universal bit allocation outperforms a JPEG-style code by over 2.5 dB. On a collection of mixed test and image data, weighted universal transform coding outperforms a single, data-optimized transform code (which gives performance almost identical to that of JPEG) by over 6 dB.     

Progressive source coding for a power constrained Gaussian channel

We consider the progressive transmission of a lossy source across a power constrained Gaussian channel using binary phase-shift keying modulation. Under the theoretical assumptions of infinite bandwidth, arbitrarily complex channel coding, and lossless transmission, we derive the optimal channel code rate and the optimal energy allocation per transmitted bit. Under the practical assumptions of a low complexity class of algebraic channel codes and progressive image coding, we numerically optimize the choice of channel code rate and the energy per bit allocation. This model provides an additional degree of freedom with respect to previously proposed schemes, and can achieve a higher performance for sources such as images. It also allows one to control bandwidth expansion or reduction