On optimal soft-decision demodulation

Wozencraft and Kennedy have suggested that the appropriate demodulator criterion of goodness is the cutoff rate of the discrete memoryless channel (DMC) created by the modulation system; the criterion of goodness adopted in this paper is the "symmetric" cutoff rate which differs from the former criterion only in that the signals are assumed equally likely. Massey's necessary condition for optimal coherent demodulation of binary signals is generalized toM-ary signals. It is shown that the optimal demodulator decision regions in likelihood space are bounded by hyperplanes. An iterative method is formulated for finding these optimal decision regions from an initial "good guess." For additive white Gaussian noise (AWGN), the corresponding optimal decision regions in signal space are bounded by hypersurfaces with hyperplane asymptotes; these asymptotes themselves bound the decision regions of a demodulator which, in several examples, is shown to be virtually optimal. In many cases, the necessary condition for demodulator optimality is also sufficient, but a counterexample to its general sufficiency is given.