Information and Entropy (Penfield)

Bits

Information is measured in bits, just as length is measured in meters and time is measured in seconds. Of course knowing the amount of information, in bits, is not the same as knowing the information itself, what it means, or what it implies. In these notes we will not consider the content or meaning of information, just the quantity.

Different scales of length are needed in different circumstances. Sometimes we want to measure length in kilometers, sometimes in inches, and sometimes in Ångströms. Similarly, other scales for information besides bits are sometimes used; in the context of physical systems information is often measured in Joules per Kelvin.

How is information quantified? Consider a situation or experiment that could have any of several possible outcomes. Examples might be flipping a coin (2 outcomes, heads or tails) or selecting a card from a deck of playing cards (52 possible outcomes). How compactly could one person (by convention often named Alice) tell another person (Bob) the outcome of such an experiment or observation?

First consider the case of the two outcomes of flipping a coin, and let us suppose they are equally likely. If Alice wants to tell Bob the result of the coin toss, she could use several possible techniques, but they are all equivalent, in terms of the amount of information conveyed, to saying either “heads” or “tails” or to saying 0 or 1. We say that the information so conveyed is one bit.

If Alice flipped two coins, she could say which of the four possible outcomes actually happened, by saying 0 or 1 twice. Similarly, the result of an experiment with eight equally likely outcomes could be conveyed with three bits, and more generally 2n outcomes with n bits. Thus the amount of information is the logarithm (to the base 2) of the number of equally likely outcomes.

Note that conveying information requires two phases. First is the “setup” phase, in which Alice and Bob agree on what they will communicate about, and exactly what each sequence of bits means. This common understanding is called the code. For example, to convey the suit of a card chosen from a deck, their code might be that 00 means clubs, 01 diamonds, 10 hearts, and 11 spades. Agreeing on the code may be done before any observations have been made, so there is not yet any information to be sent. The setup phase can include informing the recipient that there is new information. Then, there is the “outcome” phase, where actual sequences of 0 and 1 representing the outcomes are sent. These sequences are the data. Using the agreed-upon code, Alice draws the card, and tells Bob the suit by sending two bits of data. She could do so repeatedly for multiple experiments, using the same code.

After Bob knows that a card is drawn but before receiving Alice’s message, he is uncertain about the suit. His uncertainty, or lack of information, can be expressed in bits. Upon hearing the result, his uncertainty is reduced by the information he receives. Bob’s uncertainty rises during the setup phase and then is reduced during the outcome phase.