Basics of information theory

Basics of information theory


We would like to develop a usable measure of the information we get from observing the occurrence of an event having probability p. Our first reduction will be to ignore any particular features of the event, and only observe whether or not it happened. Thus we will think of an event as the observance of a symbol whose probability of occurring is p. we will thus be defining the information in terms of the probability p . A specific special case of interest is probabilities (i.e., real numbers between 0 and 1), We will
want our information measure I(p) to have several properties:
1. Information is a non-negative quantity: I(p) >=0.
2. If an event has probability 1, we get no information from the occurrence of the event: I(1) = 0.
3- If two independent events occur (whose joint probability is the product of their individual probabilities), then the information we get from observing the events is the sum of the two information:
I(p1*p2) =I(p1) +I(p2) :(This is the critical property . . . )
Some probability ideas

At various times in what follows, I may float between two notions of the probability of an event happening. The two general notions are:
1. A frequentist version of probability: In this version, we assume we have a set of possible events, each of which we assume occurs some number of times. Thus, if there are N distinct possible events (x1; x2; ; xN); no two of which can occur simultaneously, and the events occur with frequencies
(n1; n2; ; nN),