Elements of Information Theoretic Methods: Tutorial Introduction

The rudiments of information theoretic methods are introduced and companion papers dealing with solutions of some reliability problems using information theoretic approaches are preluded. Major elements of the communication system are outlined from an information processing point of view. Information is quantified following the work of Shannon. The concepts of uncertainty, self-and mutual information, and entropy are developed as seen at the encoder, channel, and decoder. Channel modeling is demonstrated using a binary symmetric channel as an example. Channel capacity is derived by maximizing transinformation. Elements of coding are described and Shannon's fundamental theorem of discrete noiseless coding is stated. The fundamental relations governing unique decipherability and irreducibility are given and demonstrated by examples. Code efficiency and redundancy are quantified and discussed as system parameters subject to tradeoff. Applications of information theoretic methods in various disciplines are discussed with emphasis on reliability and maintainability. Two unresolved reliability problem areas are identified where, potentially, information theoretic approaches may present a viable solution.