نظرية المعلومات-المحاضرة السابعة

The typical flowchart of a communication system is shown in Figure below. It applies to situations as diverse as communication between the earth and a satellite, the cellular phones, or storage within the hard disk of your computer.
Alice wants to send a message m to Bob. Let us assume that m is a M bit sequence. This message is first encoded into a longer one. The encoded message is sent through the communication channel. The output of the channel is a message y.
In a realistic channel, y is in general a string of symbols different from x.
When dealing with “information”, one of the basic goals is to transmit it reliably. In this context, “transmit” both means “transmitting some information from one point to another”, as we usually understand it, but also to “transmit” it through time; for instance to store it somewhere (to memorize it) and then retrieve it later on. In both cases however, transmission of information can, in real life, hardly be achieved in a fully reliable manner.
There always exists a risk of distortion of the transmitted information: some noise on the line, some leak of the memory or the hard disk storing the information, etc.
What effect does noise have on the transmission of messages? Several situations could be possible:
1- it is never possible to transmit any messages reliably (too much noise);
2- it is possible to transmit messages with a “reasonable” error probability;
3- it is possible to transmit messages with an error probability which is as small as we can wish for (using error correcting codes).
Channel Capacity
What do we mean when we say that A communicates with B? Source symbols from some finite