Universal codes for a class of composite sources (Corresp.)

Fixed rate universal block source coding with a fidelity criterion is considered for classes of composite sources with a finite (fixed) set of modes not an unknown switch process. In particular, it is shown that weakly minimax universal codes of all rates with respect to an arbitrary distortion measure exist for such processes.     

Efficient universal noiseless source codes

Although the existence of universal noiseless variable-rate codes for the class of discrete stationary ergodic sources has previously been established, very few practical universal encoding methods are available. Efficient implementable universal source coding techniques are discussed in this paper. Results are presented on source codes for which a small value of the maximum redundancy is achieved with a relatively short block length. A constructive proof of the existence of universal noiseless codes for discrete stationary sources is first presented. The proof is shown to provide a method for obtaining efficient universal noiseless variable-rate codes for various classes of sources. For memoryless sources, upper and lower bounds are obtained for the minimax redundancy as a function of the block length of the code. Several techniques for constructing universal noiseless source codes for memoryless sources are presented and their redundancies are compared with the bounds. Consideration is given to possible applications to data compression for certain nonstationary sources.     

Syndrome-source-coding and its universal generalization

A method of using error-correcting codes to obtain data compression, called syndrome-source-coding, is described in which the source sequence is treated as an error pattern whose syndrome forms the compressed data. It is shown that syndrome-source-coding can achieve arbitrarily small distortion with the number of compressed digits per source digit arbitrarily close to the entropy of a binary memoryless source. A "universal" generalization of syndrome-source-coding is formulated which provides robustly effective distortionless coding of source ensembles. Two examples are given comparing the performance of noiseless universal syndrome-source-coding to 1) run-length coding and 2) Lynch-Davisson-Schalkwijk-Cover universal coding for an ensemble of binary memoryless sources.     

A vector quantization approach to universal noiseless coding andquantization

A two-stage code is a block code in which each block of data is coded in two stages: the first stage codes the identity of a block code among a collection of codes, and the second stage codes the data using the identified code. The collection of codes may be noiseless codes, fixed-rate quantizers, or variable-rate quantizers. We take a vector quantization approach to two-stage coding, in which the first stage code can be regarded as a vector quantizer that “quantizes” the input data of length n to one of a fixed collection of block codes. We apply the generalized Lloyd algorithm to the first-stage quantizer, using induced measures of rate and distortion, to design locally optimal two-stage codes. On a source of medical images, two-stage variable-rate vector quantizers designed in this way outperform standard (one-stage) fixed-rate vector quantizers by over 9 dB. The tail of the operational distortion-rate function of the first-stage quantizer determines the optimal rate of convergence of the redundancy of a universal sequence of two-stage codes. We show that there exist two-stage universal noiseless codes, fixed-rate quantizers, and variable-rate quantizers whose per-letter rate and distortion redundancies converge to zero as (k/2)n ^-1 log n, when the universe of sources has finite dimension k. This extends the achievability part of Rissanen's theorem from universal noiseless codes to universal quantizers. Further, we show that the redundancies converge as O(n^-1) when the universe of sources is countable, and as O(n^-1+ϵ) when the universe of sources is infinite-dimensional, under appropriate conditions     

Time-invariant trellis encoding of ergodic discrete-time sources with a fidelity criterion

The theory of sliding-block codes (nonlinear, time-invariant, discrete-time filters) is employed to obtain general source coding theorems for ergodic sources using time-invariant trellis coding (time-invariant decoding filter and replicating trellis). The results are coupled with the theory of universal block source codes to obtain universal trellis source coding theorems for classes of sources. It is shown for a certain class of sources that the problem of designing good trellis codes is equivalent to that of simulating general random processes by filtering digital memoryless sources.     

Weighted universal image compression

We describe a general coding strategy leading to a family of universal image compression systems designed to give good performance in applications where the statistics of the source to be compressed are not available at design time or vary over time or space. The basic approach considered uses a two-stage structure in which the single source code of traditional image compression systems is replaced with a family of codes designed to cover a large class of possible sources. To illustrate this approach, we consider the optimal design and use of two-stage codes containing collections of vector quantizers (weighted universal vector quantization), bit allocations for JPEG-style coding (weighted universal bit allocation), and transform codes (weighted universal transform coding). Further, we demonstrate the benefits to be gained from the inclusion of perceptual distortion measures and optimal parsing. The strategy yields two-stage codes that significantly outperform their single-stage predecessors. On a sequence of medical images, weighted universal vector quantization outperforms entropy coded vector quantization by over 9 dB. On the same data sequence, weighted universal bit allocation outperforms a JPEG-style code by over 2.5 dB. On a collection of mixed test and image data, weighted universal transform coding outperforms a single, data-optimized transform code (which gives performance almost identical to that of JPEG) by over 6 dB.     

The design of joint source and channel trellis waveform coders

The generalized Lloyd algorithm is applied to the design of joint source and channel trellis waveform coders to encode discrete-time continuous-amplitude stationary and ergodic sources operating over discrete memoryless noisy channels. Experimental results are provided for independent and autoregressive Gaussian sources, binary symmetric channels, and absolute error and squared error distortion measures. Performance of the joint codes is compared with the tandem combination of a trellis source code and a trellis channel code on the independent Gaussian source using the squared error distortion measure operating over an additive white Gaussian noise channel. It is observed that the jointly optimized codes achieve performance close to or better than that of separately optimized tandem codes of the same constraint length. Performance improvement via a predictive joint source and channel trellis code is demonstrated for the autoregressive Gaussian source using the squared error distortion measure.     

Strong universal source coding subject to a rate-distortion constraint

The following problem in universal source coding is explored. Some members of a class of sources have various constraints on the maximum rate at which they may be encoded, and thc remainder have various constraints on the maximum distortion that may result from encoding. It is desired to find a universal code that will adapt its performance so that whatever source in the class is encoded, the resulting performance meets the constraint and is optimal in the rate-distortion function sense for that particular source. It is shown that such codes exist when the class is totally bounded and the constraints are uniformly continuous in an appropriate sense. A key result shows that the value of any uniformly continuous function on a totally bounded class can be uniformly well estimated from observations of the output of any source in     

Variable rate coding for nonergodic sources and classes of ergodic sources subject to a fidelity constraint

This paper is concerned with two problems in the theory of source coding subject to a maximum average distortion constraint. The first problem involves the coding of a nonergodic discrete time source and the second involves coding for a class of ergodic discrete time sources. Coding theorems are given for both of these situations for very general source alphabets. The codes that are obtained here will, in general, be variable length codes so that average code rate and average distortion are the measures of performance.     

New results on coding of stationary nonergodic sources

Two results on the coding of stationary nonergodic sources are presented. The first is a source coding theorem stating that there exist variable-rate codes with performance arbitrarily close to the rate-distortion function of the stationary nonergodic source. The second is a converse information transmission theorem. It is shown that the distortion which results when the source is transmitted across a channel with capacityCis no less than the least distortion achievable by fixed-rate codes with rateC.     

Fixed-rate universal lossy source coding and rates of convergencefor memoryless sources

A fixed-rate universal lossy coding scheme is introduced for independent and identically distributed (i.i.d.) sources. It is shown for finite alphabet sources and arbitrary single letter distortion measures that as the sample size n grows the expected distortion obtained using this universal scheme converges to Shannon's distortion rate function D(R) at a rate O(log n/n). The scheme can be extended to universal quantization of real i.i.d sources subject to a squared error criterion. It is shown in this case that the per-letter distortion converges to D(R) at a rate O(√(log n/n)) both in expectation and almost surely for any real-valued bounded i.i.d. source     

Tree encoding of stationary ergodic sources with a fidelity criterion

Tree source coding theorems with a single letter fidelity criterion are proved for stationary ergodic sources using tree codes with a fixed branch length. The higher-than-exponential convergence of distortions is shown for binary symmetric sources and Hamming distortion measure when an excess in rate is allowed.     

Source Coding Using Families of Universal Hash Functions

This correspondence is concerned with new connections between source coding and two kinds of families of hash functions known as the families of universal hash functions and N-strongly universal hash functions, where N ges 2 is an integer. First, it is pointed out that such families contain classes of well-known source codes such as bin codes and linear codes. Next, performance of a source coding scheme using either of the two kinds of families is evaluated. An upper bound on the expectation of the decoding error probability is obtained for each family. The expectation of the decoding error probability is analyzed in detail for the cases of discrete memoryless sources and sources without the memoryless assumption under a certain class of decoders.     

A universal algorithm for sequential data compression

A universal algorithm for sequential data compression is presented. Its performance is investigated with respect to a nonprobabilistic model of constrained sources. The compression ratio achieved by the proposed universal code uniformly approaches the lower bounds on the compression ratios attainable by block-to-variable codes and variable-to-block codes designed to match a completely specified source.     

Universal variable-to-fixed length source codes

A universal variable-to-fixed length algorithm for binary memoryless sources which converges to the entropy of the source at the optimal rate is known. We study the problem of universal variable-to-fixed length coding for the class of Markov sources with finite alphabets. We give an upper bound on the performance of the code for large dictionary sizes and show that the code is optimal in the sense that no codes exist that have better asymptotic performance. The optimal redundancy is shown to be H log log M/log M where H is the entropy rate of the source and M is the code size. This result is analogous to Rissanen's (1984) result for fixed-to-variable length codes. We investigate the performance of a variable-to-fixed coding method which does not need to store the dictionaries, either at the coder or the decoder. We also consider the performance of both these source codes on individual sequences. For individual sequences we bound the performance in terms of the best code length achievable by a class of coders. All the codes that we consider are prefix-free and complete     

Universal source coding of finite alphabet sources via composition classes

The universal source coding theorem for stationary sources states that, by using a preselected code with sufficiently large block length, any stationary source can be encoded arbitrarily close to the optimum performance theoretically attainable for the source. The code construction typically used to prove this theorem offers no performance guarantees for nonstationary sources. A code construction (based on composition classes) is described which is shown to be universal for stationary sources. In addition, this code construction guarantees that the distortion for any block in the sequence will not exceed a maximum value determined by its composition.     

Universal noiseless coding

Universal coding is any asymptotically optimum method of block-to-block memoryless source coding for sources with unknown parameters. This paper considers noiseless coding for such sources, primarily in terms of variable-length coding, with performance measured as a function of the coding redundancy relative to the per-letter conditional source entropy given the unknown parameter. It is found that universal (i.e., zero redundancy) coding in a weighted sense is possible if and only if the per-letter average mutual information between the parameter space and the message space is zero. Universal coding is possible in a maximin sense if and only if the channel capacity between the two spaces is zero. Universal coding is possible in a minimax sense if and only if a probability mass function exists, independent of the unknown parameter, for which the relative entropy of the known conditional-probability mass-function is zero. Several examples are given to illustrate the ideas. Particular attention is given to sources that are stationary and ergodic for any fixed parameter although the whole ensemble is not. For such sources, weighted universal codes always exist if the alphabet is finite, or more generally if the entropy is finite. Minimax universal codes result if an additional entropy stability constraint is applied. A discussion of fixed-rate universal coding is also given briefly with performance measured by a probability of error.     

Tree coding of image subbands

The authors consider the encoding of image subbands with a tree code that is asymptotically optimal for Gaussian sources and the mean squared error (MSE) distortion measure. They first prove that optimal encoding of ideally filtered subbands of a Gaussian image source achieves the rate distortion bound for the MSE distortion measure. The optimal rate and distortion allocation among the subbands is a by-product of this proof. A bound is derived which shows that subband coding is closer than full-band coding to the rate distortion bound for a finite length sequence. The tree codes are then applied to encode the image subbands, both nonadaptively and adaptively. Since the tree codes are stochastic and the search of the code tree is selective, a relatively few reproduction symbols may have an associated squared error a hundred times larger than the target for the subband. Correcting these symbols through a postcoding procedure improves the signal-to-noise ratio and visual quality significantly, with a marginal increase in total rate.     

The redundancy of source coding with a fidelity criterion .II.Coding at a fixed rate level with unknown statistics

The redundancy problem of universal lossy source coding at a fixed rate level is considered. Under some condition on the single-letter distortion measure, which implies that the cardinality K of the reproduction alphabet is not greater than the cardinality J of the source alphabet, it is shown that the redundancy of universally coding memoryless sources p by nth-order block codes of rate R goes like |(∂/∂R)d(p,R)|Kln n/2n+o(ln n/n) for all memoryless sources p except a set whose volume goes to 0 as the block length n goes to infinity, where d(p,R) denotes the distortion rate function of p. Specifically, for any sequence {C_n}_n=1^∞ of block codes, where C_n is an nth-order block code at the fixed rate R, and any ϵ>0, the redundancy D_n(C _n,p) of C_n for p is greater than or equal to |(∂/∂R)d(p,R)|(K-ϵ)ln n/2n for all p satisfying some regular conditions except a set whose volume goes to 0 as n→∞. On the other hand, there exists a sequence {C_n}_n=1^∞ of block codes at the rate R such that for any p satisfying some regular conditions, the super limit of D_n(C_n,p)|(ln n/n) is less than or equal to |(∂/∂R)d(p,R)|K/2