Hypothesis testing with multiterminal data compression

The multiterminal hypothesis testingH: XYagainstH̄: X̄Ȳis considered whereX^{n} (X̄^{n})andY^{n} (Ȳ^{n})are separately encoded at ratesR_{1}andR_{2}, respectively. The problem is to determine the minimumbeta_{n}of the second kind of error probability, under the condition that the first kind of error probabilityalpha_{n} leq epsilonfor a prescribed0 < epsilon < 1. A good lower boundtheta_{L}(R_{1}, R_{2})on the power exponenttheta (R_{1}, R_{2},epsilon)= lim inf_{n rightarrow infty}(-1/n log beta_{n})is given and several interesting properties are revealed. The lower bound is tighter than that of Ahlswede and Csiszár. Furthermore, in the special case of testing against independence, this bound turns out to coincide with that given by them. The main arguments are devoted to the special case withR_{2} = inftycorresponding to full side information forY^{n}(Ȳ^{n}). In particular, the compact solution is established to the complete data compression cases, which are useful in statistics from the practical point of view.     

On certain projective geometry codes (Corresp.)

LetVbe an(n, k, d)binary projective geometry code withn = (q^{m}-1)/(q - 1), q = 2^{s}, andd geq [(q^{m-r}-1)/(q - 1)] + 1. This code isr-step majority-logic decodable. With reference to the GF(q^{m}) = {0, 1, alpha , alpha^{2} , cdots , alpha^{n(q-1)-1} }, the generator polynomialg(X), ofV, hasalpha^{nu}as a root if and only ifnuhas the formnu = i(q - 1)andmax_{0 leq l < s} W_{q}(2^{l} nu) leq (m - r - 1)(q - 1), whereW_{q}(x)indicates the weight of the radix-qrepresentation of the numberx. LetSbe the set of nonzero numbersnu, such thatalpha^{nu}is a root ofg(X). LetC_{1}, C_{2}, cdots, C_{nu}be the cyclotomic cosets such thatSis the union of these cosets. It is clear that the process of findingg(X)becomes simpler if we can find a representative from eachC_{i}, since we can then refer to a table, of irreducible factors, as given by, say, Peterson and Weldon. In this correspondence it was determined that the coset representatives for the cases ofm-r = 2, withs = 2, 3, andm-r=3, withs=2.     

Sequences achieving the boundary of the entropy region for a two-source are virtually memoryless (Corresp.)

For a joint distribution{rm dist}(X,Y), the functionT(t)=min { H(Y|U): I(U wedge Y|X)=O, H(X|U)geq t}is an important characteristic. It equals the asymptotic minimum of(1/n)H(Y^{n})for random pairs of sequences(X^{n}, Y^{n}), wherefrac{1}{n} sum ^{n}_{i=1}{rm dist} X_{i} sim {rm dist} X, {rm dist} Y^{n}|X^{n} = ({rm dist} Y|X)^{n}, frac{1}{n}H(X^{n})geq t.We show that if, for(X^{n}, Y^{n})as given, the rate pair[(1/n)H(X^{n}),(1/n)H(Y^{n})]approaches the nonlinear part of the curve(t,T(t)), then the sequenceX^{n}is virtually memoryless. Using this, we determine some extremal sections of the rate region of entropy characterization problems and find a nontrivial invariant for weak asymptotic isomorphy of discrete memoryless correlated sources.     

Multiple-burst error-correcting cyclic product codes (Corresp.)

LetCbe the cyclic product code ofpsingle parity check codes of relatively prime lengthsn_{1}, n_{2},cdots , n_{p} (n_{1} < n_{2} < cdots < n_{p}). It is proven thatCcan correct2^{P-2}+2^{p-3}-1bursts of lengthn_{1}, andlfloor(max{p+1, min{2^{p-s}+s-1,2^{p-s}+2^{p-s-1}}}-1)/2rfloorbursts of lengthn_{1}n_{2} cdots n_{s} (2leq s leq p-2). Forp=3this means thatCis double-burst-n_{1}-correcting. An efficient decoding algorithm is presented for this code.     

Higher dimensional orthogonal designs and applications

The concept of orthogonal design is extended to higher dimensions. A properg-dimensional design[d_{ijk cdots upsilon}]is defined as one in which all parallel(g-1)-dimensional layers, in any orientation parallel to a hyper plane, are uncorrelated. This is equivalent to the requirement thatd_{ijk cdots upsilon} in {0, pm x_{1}, cdots , pm x_{t} }, wherex_{1}, cdots , x_{t}are commuting variables, and thatsum_{p} sum_{q} sum_{r} cdots sum_{y} d_{pqr cdots ya} d_{pqr cdots yb} = left( sum_{t} s_{i}x_{i}^{2} right)^{g-1} delta ab,where(s{1}, cdots , s{t})are integers giving the occurrences ofpm x_{1}, cdots , pm x_{t}in each row and column (this is called the type(s_{1}, cdot ,s_{t})^{g-1})and(pqr cdots yz)represents all permutations of(ijk cdots upsilon). This extends an idea of Paul J. Shlichta, whose higher dimensional Hadamard matrices are special cases withx_{1}, cdots , x_{t} in {1,- 1}, (s_{1}, cdots, s_{t})=(g), and(sum_{t}s_{i}x_{i}^{2})=g. Another special case is higher dimensional weighing matrices of type(k)^{g}, which havex_{1}, cdots , x_{t} in {0,1,- 1}, (s_{1}, cdots, s_{t})=(k), and(sum_{t}s_{i}x_{i}^{2})=k. Shlichta found properg-dimensional Hadamard matrices of size(2^{t})^{g}. Proper orthogonal designs of type     

Equipartition laws of thermal noise

Classically, the thermal noise in electricalRCcircuits andLCRseries circuits is governed by the equipartition lawfrac{1}{2}overline{CV^{2}} = frac{1}{2}kT, whereV(t)is the noise voltage developed acrossC. When quantum effects are taken into account, the equipartition law no longer holds forRCcircuits, although an equipartition law can be deemed for the measured mean square noise voltage under certain conditions. InLCRseries circuits the equipartition lawfrac{1}{2}overline{CV^{2}} = frac{1}{2}kT, changes intofrac{1}{2}overline{CV^{2}} = frac{1}{2}bar{E}(f_{0})for high-Qtuned circuits, wherebar{E}(f_{0})is the average energy of a harmonic oscillator tuned at the tuning frequency of the tuned circuit.     

A dichotomy of functionsF(X, Y)of correlated sources(X, Y)

Consider separate encoding of correlated sourcesX^{n}=(X_{l}, cdots ,X_{n}), Y^{n} = (Y_{l}, cdots ,Y_{n})for the decoder to reliably reproduce a function{F(X_{i}, Y_{i})}^{n}_{i=1}. We establish the necessary and sufficient condition for the set of all achievable rates to coincide with the Slepian-Wolf region whenever the probability densityp(x,y)is positive for all(x,y).     

Successive encoding of correlated sources

The encoding of a discrete memoryless multiple source{( X_{i}, Y_{i})}_{i=1}^{infty}for reconstruction of a sequence{Z_{i}}_{i=1}^{infty}}, withZ_{i} = F( X_{i}, Y_{i}); i = 1,2, cdotsis considered. We require that the encoding should be such that{X_{i}}_{i=1}^{infty}is encoded first without any consideration of{Y_{i}}_{i=1}^{infty}, while in a second part of the encoding, this latter sequence is encoded based on knowledge of the outcome of the first encoding. The resulting scheme is called successive encoding. We find general outer and inner bounds for the corresponding set of achievable rates along with a complete single letter characterization for the special caseH( X_{i}|Z_{i}, Y_{i}) = 0. Comparisons with the Slepian-Wolf problem and the Ahlswede-Korner-Wyner side information problem are carried out.     

Distribution-free performance bounds for potential function rules

In the discrimination problem the random variabletheta, known to take values in{1, cdots ,M}, is estimated from the random vectorX. All that is known about the joint distribution of(X, theta)is that which can be inferred from a sample(X_{1}, theta_{1}), cdots ,(X_{n}, theta_{n})of sizendrawn from that distribution. A discrimination nde is any procedure which determines a decisionhat{ theta}forthetafromXand(X_{1}, theta_{1}) , cdots , (X_{n}, theta_{n}). For rules which are determined by potential functions it is shown that the mean-square difference between the probability of error for the nde and its deleted estimate is bounded byA/ sqrt{n}whereAis an explicitly given constant depending only onMand the potential function. TheO(n ^{-1/2})behavior is shown to be the best possible for one of the most commonly encountered rules of this type.     

Coding the vertexes of a graph

Given a graphGofnnodes. We wish to assign to each nodei(i = 1, 2, cdots n)a unique binary codec_{i}of lengthmsuch that, if we denote the Hannuing distance betweenc_{i}andc_{j}asH(c_{i}, c_{j}), thenH(c_{i}, c_{j})leq Tif nodesiandjare adjacent (i.e., connected by a single branch), andH(c_{i}, c_{j}) geq T+1otherwise. If such a code exists, then we say thatGis doable for the value ofTand tn associated with this code. In this paper we prove various properties relevent to these codes. In particular we prove 1) that for every graphGthere exists anmandTsuch thatGis doable, 2) for every value ofTthere exists a graphG̃which is notTdoable, 3) ifGisT'doable, then it isT'+ 2pdoable forp = 0, 1, 2, cdots, and is doable for allT geq 2T'ifT'is odd, and is doable for allT geq 2T' + 1ifT'is even. In theory, the code can be synthesized by employing integer linear programming where eitherTand/ormcan be minimized; however, this procedure is computationally infeasible for values ofnandmin the range of about10or greater.     

Some cross correlation properties for distorted signals

For a nondecreasing distortion characteristicphi(cdot)and a given signalx(cdot), the "cross correlation" function defined byR_{phi} (tau) triangleq int_{-infty}^{infty} phi[x(t)]x(t - tau) dtis shown to satisfy the inequalityR_{phi}(tau) leq R_{phi}(0), for alltau, generalizing an earlier result of Richardson that requiredphi(cdot)to be continuous and strictly increasing. The methods of the paper also show that, under weak conditions, begin{equation} R_{phi,psi}(tau) triangleq int_{-infty}^{infty} phi[x(t)]psi[x(t - tau)] dt leq R_{phi,psi}(0) end{equation} whenpsiis strictly increasing andphiis nondecreasing. In the case of hounded signals (e.g., periodic functions), the appropriate cross correlation function is begin{equation} mathcal{R}_{phi,psi}(tau} triangleq lim_{T rightarrow infty} (2T)^{-l} int_{-T}^T phi[x(t)]psi[x(t - tau)] dt. end{equation} For this case it is shown thatmathcal{R}_{phi,psi} (tau) leq mathcal{R}_{phi,psi}(0)for any nondecreasing (or nonincreasing) distortion functionsphiandpsi. The result is then applied to generalize an inequality on correlation functions for periodic signals due to Prosser. Noise signals are treated and inequalities of a similar nature are obtained for ensemble-average cross correlation functions under suitable hypotheses on the statistical properties of the noise. Inequalities of this type are the basis of a well-known method of estimating the unknown time delay of an observed signal. The extension to nondecreasing discontinuous distortion functions allows the use of hard limiting or quantization to facilitate the cross correlation calculation.     

Bounds on the redundancy of Huffman codes (Corresp.)

New upper bounds on the redundancy of Huffman codes are provided. A bound that for2/9 leq P_{1} leq 0.4is sharper than the bound of Gallager, when the probability of the most likely source letterP_{1}is the only known probability is presented. The improved bound is the tightest possible for1/3 leq P_{1} leq 0.4. Upper bounds are presented on the redundancy of Huffman codes when the extreme probabilitiesP_{1}andP_{N}are known.     

Repeated use of codes which detect deception (Corresp.)

A senderAwants to sendNmessagesx_{i} = 1 , ldots ,N, chosen from a set containingMdifferent possible messages,M > N, to a receiverB. Everyx_{i}has to pass through the hands of a dishonest messengerC. ThereforeAandBagree on a mathematical transformationfand a secret parameter, or keyk, that will be used to produce the authenticatory_{i} = f(x_{i} , k ), which is sent together withx. The key is chosen at random from a set ofLelements.Cknowsfand can find all elements in the setG(x_{i},y_{i}) = {k|f(x_{i}, k) = y_{i}}given enough time and computer resources.Cwants to changex_{i} into x^{prime}withoutBsuspecting. This means thatCmust find the new anthenticatory^{prime} = f(x^{prime} , k). SinceG(x_{i},y_{i})can be found for any(x_{i},y_{i}), it is obvious thatCwill always succeed unlessG(x_{i},y_{i})contains more than one element. Here it is proved that the average probability of success forCis minimized if (a)G(x_{i}, y_{i})containsL^{(N-1)/N}elements and (b) each new known pair(x_{j}, y_{j})will diminish this set of solutions by a factor ofL^{-l/N}. The minimum average probability will then beL^{-l/N}.     

Recursive computations for the optimal tracking of time-varying parameters (Corresp.)

Recursive relations are given for updating the conditional densityp(theta_{k} | X_{k-1}, cdots X_{1})(also forp(theta_{k} | X_{k}, cdots, X_{1})), wheretheta_{k}is a parameter of the density ofX_{k}. The observationsX_{1}, X_{2}, cdotsare assumed to be conditionally independent (i.e., for known parameters), and the sequence of time-varying parameterstheta_{1}, theta_{2}, cdotsconstitutes a Markov-M sequence. The result requires the storage of an intermediate function of(theta_{k-1}, cdots , theta_{k-M}).     

Capacity theorems for the relay channel

A relay channel consists of an inputx_{l}, a relay outputy_{1}, a channel outputy, and a relay senderx_{2}(whose transmission is allowed to depend on the past symbolsy_{1}. The dependence of the received symbols upon the inputs is given byp(y,y_{1}|x_{1},x_{2}). The channel is assumed to be memoryless. In this paper the following capacity theorems are proved. 1)Ifyis a degraded form ofy_{1}, thenC : = : max !_{p(x_{1},x_{2})} min ,{I(X_{1},X_{2};Y), I(X_{1}; Y_{1}|X_{2})}. 2)Ify_{1}is a degraded form ofy, thenC : = : max !_{p(x_{1})} max_{x_{2}} I(X_{1};Y|x_{2}). 3)Ifp(y,y_{1}|x_{1},x_{2})is an arbitrary relay channel with feedback from(y,y_{1})to bothx_{1} and x_{2}, thenC: = : max_{p(x_{1},x_{2})} min ,{I(X_{1},X_{2};Y),I ,(X_{1};Y,Y_{1}|X_{2})}. 4)For a general relay channel,C : leq : max_{p(x_{1},x_{2})} min ,{I ,(X_{1}, X_{2};Y),I(X_{1};Y,Y_{1}|X_{2}). Superposition block Markov encoding is used to show achievability ofC, and converses are established. The capacities of the Gaussian relay channel and certain discrete relay channels are evaluated. Finally, an achievable lower bound to the capacity of the general relay channel is established.     

Maximum entropy and conditional probability

It is well-known that maximum entropy distributions, subject to appropriate moment constraints, arise in physics and mathematics. In an attempt to find a physical reason for the appearance of maximum entropy distributions, the following theorem is offered. The conditional distribution ofX_{l}given the empirical observation(1/n)sum^{n}_{i}=_{l}h(X_{i})=alpha, whereX_{1},X_{2}, cdotsare independent identically distributed random variables with common densitygconverges tof_{lambda}(x)=e^{lambda^{t}h(X)}g(x)(Suitably normalized), wherelambdais chosen to satisfyint f_{lambda}(x)h(x)dx= alpha. Thus the conditional distribution of a given random variableXis the (normalized) product of the maximum entropy distribution and the initial distribution. This distribution is the maximum entropy distribution whengis uniform. The proof of this and related results relies heavily on the work of Zabell and Lanford.     

An information - theoretic proof of Hadamard's inequality (Corresp.)

Hadamard's inequality follows immediately from inspection of both sides of the entropy inequalityh(X_{1}, X_{2},, cdots, X_{n})leq sum h(X_{i}), when(X_{l}, X_{2},cdots, X_{n})is multivariate normal.     

On theepsilon-entropy and the rate-distortion function of certain non-Gaussian processes

Letxi = {xi(t), 0 leq t leq T}be a process with covariance functionK(s,t)andE int_0^T xi^2(t) dt < infty. It is proved that for everyvarepsilon > 0thevarepsilon-entropyH_{varepsilon}(xi)satisfies begin{equation} H_{varepsilon}(xi_g) - mathcal{H}_{xi_g} (xi) leq H_{varepsilon}(xi) leq H_{varepsilon}(xi_g) end{equation} wherexi_gis a Gaussian process with the covarianeeK(s,t)andmathcal{H}_{xi_g}(xi)is the entropy of the measure induced byxi(in function space) with respect to that induced byxi_g. It is also shown that ifmathcal{H}_{xi_g}(xi) < inftythen, asvarepsilon rightarrow 0begin{equation} H_{varepsilon}(xi) = H_{varepsilon}(xi_g) - mathcal{H}_{xi_g}(xi) + o(1). end{equation} Furthermore, ff there exists a Gaussian processg = { g(t); 0 leq t leq T }such thatmathcal{H}_g(xi) < infty, then the ratio betweenH_{varepsilon}(xi)andH_{varepsilon}(g)goes to one asvarepsilongoes to zero. Similar results are given for the rate-distortion function, and some particular examples are worked out in detail. Some cases for whichmathcal_{xi_g}(xi) = inftyare discussed, and asymptotic bounds onH_{varepsilon}(xi), expressed in terms ofH_{varepsilon}(xi_g), are derived.