Variable-rate universal block source coding subject to a fidelity constraint

Variable-rate universal source codes are data compression schemes that are optimum for the coding of a collection of sources (e.g., a source with unknown parameters) subject to a fixed average distortion constraint. Existence of variable-rate universal source codes is demonstrated for very general classes of sources and distortion measures.     

Generalized Tunstall codes for sources with memory

Tunstall codes are variable-to-fixed length codes that maximize the expected number of source letters per dictionary string for discrete, memoryless sources. We analyze a generalization of Tunstall coding to sources with memory and demonstrate that as the dictionary size increases, the number of code letters per source symbol comes arbitrarily close to the minimum among all variable-to-fixed length codes of the same size. We also find the asymptotic relationship between the dictionary size and the average length of a dictionary entry     

New results on coding of stationary nonergodic sources

Two results on the coding of stationary nonergodic sources are presented. The first is a source coding theorem stating that there exist variable-rate codes with performance arbitrarily close to the rate-distortion function of the stationary nonergodic source. The second is a converse information transmission theorem. It is shown that the distortion which results when the source is transmitted across a channel with capacityCis no less than the least distortion achievable by fixed-rate codes with rateC.     

Permutation codes for sources

Source encoding techniques based on permutation codes are investigated. For a broad class of distortion measures it is shown that optimum encoding of a source permutation code is easy to instrument even for very long block lengths. Also, the nonparametric nature of permutation encoding is well suited to situations involving unknown source statistics. For the squared-error distortion measure a procedure for generating good permutation codes of a given rate and block length is described. The performance of such codes for a memoryless Gaussian source is compared both with the rate-distortion function bound and with the performance of various quantization schemes. The comparison reveals that permutation codes are asymptotically ideal for small rates and perform as well as the best entropy-coded quantizers presently known for intermediate rates. They can be made to compare favorably at high rates, too, provided the coding delay associated with extremely long block lengths is tolerable.     

Optimal encoding of discrete-time continuous-amplitude memoryless sources with finite output alphabets

A rate-distortion theory is introduced for the optimal encoding of stationary memoryless continuous-amplitude sources with a single-letter distortion measure and reproduction alphabets of a given finite size. The theory arises from a judicious approximation of the original continuous-input discrete-output problem by one with discrete input and output. A size-constrained output alphabet rate-distortion function is defined, its coding significance is established by coding theorems, and a convergent algorithm is presented for its evaluation. The theory is applied to Gaussian sources with squared-error distortion measure. Using the algorithm for the calculation of the new rate-distortion function in this case establishes the existence of codes which attain almost any desired rate between the rate-distortion bound and the optimum entropy-coded quantizer. Furthermore, one can closely approach the rate-distortion limit with a surprisingly small number of output levels. The calculation furnishes optimal output levels, output level probabilities, and other parameters necessary for a trellis coding simulation. The search algorithm represents the first use for asymmetric sources and distortion measures of a variation of a single stack algorithm proposed by Gallager. Carrying out the simulation at a rate of 1 bit per source symbol, codes are found with 4 and 64 output levels which attain distortions smaller than that of an optimum quantizer and close to the rate-distortion bound. Furthermore, these codes attain comparable or better performance with far less search effort than previous attempts with a continuous output alphabet.     

Time-invariant trellis encoding of ergodic discrete-time sources with a fidelity criterion

The theory of sliding-block codes (nonlinear, time-invariant, discrete-time filters) is employed to obtain general source coding theorems for ergodic sources using time-invariant trellis coding (time-invariant decoding filter and replicating trellis). The results are coupled with the theory of universal block source codes to obtain universal trellis source coding theorems for classes of sources. It is shown for a certain class of sources that the problem of designing good trellis codes is equivalent to that of simulating general random processes by filtering digital memoryless sources.     

Adaptive buffer- instrumented entropy-coded quantizer performance for memoryless sources

Variable-length codes can be used in entropy coding the outputs of an optimum entropy-constrained quantizer. Transmitting these codes over a synchronous channel; however, requires a buffer connecting the entropy coder to the channel. In a practical application, this buffer is of finite size and hence might overflow or undertow. To alleviate this difficulty, we use an adaptive scheme in which the quantizer parameters are changed successively according to the state of the buffer. Rate-distortion performance of optimum entropy-constrained quantizers in conjunction with this adaptive scheme is studied for the class of generalized Gaussian sources. It is demonstrated through simulations that the overflow/ undertow problem can be practically eliminated at the cost of a negligible increase in average distortion. Furthermore, it is shown that the efficiency of this system is more pronounced at high rates and for more broadtailed source densities. Easily computable upper and lower bounds on the average distortion of the adaptive system are developed.     

Convergence of distortion for block coding of symmetric modular sources (Corresp.)

We define a symmetric modular source to be a source-user pair consisting of an equiprobable discrete memoryless source together with a modular context-dependent fidelity criterion. For this source-user pair we establish the existence of block codes of fixed rateRgreater thanR(D)with average distortion that decreases towardsDat a doubly exponential rate in block length.     

A transform tree code for stationary Gaussian sources

A new tree code is introduced for discrete-time stationary Gaussian sources with hounded, integrable power spectra and the squared-error distortion measure. The codewords in the tree are reconstructions of Karhunen-Loève transforms of the source words. The branching factor and the number of code letters per branch may vary with level in the tree. A theorem that guarantees the existence of an optimal code for any code rate using such a tree is proved. The proof uses the random coding argument in conjunction with a theorem on survival of a branching process with random environment. A suboptimal but computationally affordable realization of the theorem's coding technique was used for encoding simulations for six autoregressive sources at rates of1.0, 0.50, 0.25, and0.10bits per source symbol. The average distortion results were generally within1dB of the distortion-rate bound but varied widely depending on the source and rate. The results were compared with those for transform quantization simulations for the same sources and rates. The tree code always performed better but only by an average of0.44dB all sources and rates. Longer source blocks and more intensive search would certainly improve the performance of the tree codes, but at the expense of extra computation and storage.     

A coding theorem for time-discrete analog data sources

The main result of the paper is a coding theorem for time-discrete sources with a fidelity criterion on average distortion. This result applies to a subclass of the ergodic sources (stationary sources having a strong mixing property) and to the cases in which source samples are continuously distributed (analog) quantities or discrete symbols. An intuitive definition of the information content of data samples for this class of sources is then formally established and is shown to be exactly the same as Shannon's definition.     

Tree encoding of stationary ergodic sources with a fidelity criterion

Tree source coding theorems with a single letter fidelity criterion are proved for stationary ergodic sources using tree codes with a fixed branch length. The higher-than-exponential convergence of distortions is shown for binary symmetric sources and Hamming distortion measure when an excess in rate is allowed.     

A rate and distortion analysis of chain codes for line drawings

The performance of chain codes for line drawings in the framework of source coding with a fidelity criterion is investigated. A formal characterization is given for a broad class of chain codes that includes the generalized chain codes of Freeman and the convex quantizers of Koplowitz. Measures of distortion (fidelity) and rate (efficiency) are introduced. The distortion and rate are calculated for several types of chain codes applied to straight and slowly curving lines. Several statistical models for straight lines are introduced and the average performance of chain codes for these models is investigated.     

Multidimensional incremental parsing for universal source coding

A multidimensional incremental parsing algorithm (MDIP) for multidimensional discrete sources, as a generalization of the Lempel-Ziv coding algorithm, is investigated. It consists of three essential component schemes, maximum decimation matching, hierarchical structure of multidimensional source coding, and dictionary augmentation. As a counterpart of the longest match search in the Lempel-Ziv algorithm, two classes of maximum decimation matching are studied. Also, an underlying behavior of the dictionary augmentation scheme for estimating the source statistics is examined. For an m-dimensional source, m augmentative patches are appended into the dictionary at each coding epoch, thus requiring the transmission of a substantial amount of information to the decoder. The property of the hierarchical structure of the source coding algorithm resolves this issue by successively incorporating lower dimensional coding procedures in the scheme. In regard to universal lossy source coders, we propose two distortion functions, the local average distortion and the local minimax distortion with a set of threshold levels for each source symbol. For performance evaluation, we implemented three image compression algorithms based upon the MDIP; one is lossless and the others are lossy. The lossless image compression algorithm does not perform better than the Lempel-Ziv-Welch coding, but experimentally shows efficiency in capturing the source structure. The two lossy image compression algorithms are implemented using the two distortion functions, respectively. The algorithm based on the local average distortion is efficient at minimizing the signal distortion, but the images by the one with the local minimax distortion have a good perceptual fidelity among other compression algorithms. Our insights inspire future research on feature extraction of multidimensional discrete sources.     

Variable to fixed-length codes for Markov sources

Petry's efficient and optimal variable to fixed-length source code for discrete memoryless sources was described by Schalkwijk. By extending this coding technique we are able to give an algorithm for Markov sources that is easy to implement. We can bound the loss of efficiency as a function of the code complexity and the mismatch between the source and the code. Rates arbitrarily close to the source entropy are shown to be achievable. In this sense the codes introduced are optimal.     

Linear codes for sources and source networks: Error exponents, universal coding

For Slepian-Wolf source networks, the error exponents obtained by Körner,Marton, and the author are shown to be universally attainable by linear codes also. Improved exponents are derived for linear codes with "large rates." Specializing the results to simple discrete memoryless sources reveals their relationship to the random coding and expurgated bounds for channels with additive noise. One corollary is that there are universal linear codes for this class of channels which attain the random coding error exponent for each channel in the class. The combinatorial approach of Csiszár-Körner-Marton is used. In particular, all results are derived from a lemma specifying good encoders in terms of purely combinatorial properties.     

Information bounds of the Fano-Kullback type

A large class of lower bounds relating to the performance of hypothesis testers, channel codes, and source compression codes is developed. These are extensions of Fano's inequality on the one hand, and of the discrimination inequality of Kullback on the other. The hypothesis testing and channel coding bounds are interesting primarily for small blocklengths and, in general, are asymptotically inferior to the well-known exponentially decreasing bounds. The source compression results include new proofs of converse coding theorems. A lower bound is given to the probability that a source produces an output block which cannot be encoded within a desired maximum distortion.     

Source coding exponents for zero-delay coding with finite memory

Fundamental limits on the source coding exponents (or large deviations performance) of zero-delay finite-memory (ZDFM) lossy source codes are studied. Our main results are the following. For any memoryless source, a suitably designed encoder that time-shares (at most two) memoryless scalar quantizers is as good as any time-varying fixed-rate ZDFM code, in that it can achieve the fastest exponential rate of decay for the probability of excess distortion. A dual result is shown to apply to the probability of excess code length, among all fixed-distortion ZDFM codes with variable rate. Finally, it is shown that if the scope is broadened to ZDFM codes with variable rate and variable distortion, then a time-invariant entropy-coded memoryless quantizer (without time sharing) is asymptotically optimal under a "fixed-slope" large-deviations criterion (introduced and motivated here in detail) corresponding to a linear combination of the code length and the distortion. These results also lead to single-letter characterizations for the source coding error exponents of ZDFM codes.     

Progressive transmission of images over memoryless noisy channels

An embedded source code allows the decoder to reconstruct the source progressively from the prefixes of a single bit stream. It is desirable to design joint source-channel coding schemes which retain the capability of progressive reconstruction in the presence of channel noise or packet loss. Here, we address the problem of joint source-channel coding of images for progressive transmission over memoryless bit error or packet erasure channels. We develop a framework for encoding based on embedded source codes and embedded error correcting and error detecting channel codes. For a target transmission rate, we provide solutions and an algorithm for the design of optimal unequal error/erasure protection. Three performance measures are considered: the average distortion, the average peak signal-to-noise ratio, and the average useful source coding rate. Under the assumption of rate compatibility of the underlying channel codes, we provide necessary conditions for progressive transmission of joint source-channel codes. We also show that the unequal error/erasure protection policies that maximize the average useful source coding rate allow progressive transmission with optimal unequal protection at a number of intermediate rates     

Universal codes for a class of composite sources (Corresp.)

Fixed rate universal block source coding with a fidelity criterion is considered for classes of composite sources with a finite (fixed) set of modes not an unknown switch process. In particular, it is shown that weakly minimax universal codes of all rates with respect to an arbitrary distortion measure exist for such processes.