The Exercise

Toss a coin and note which side lands up; roll a die and note the face that shows. Before the toss or the roll the outcome is uncertain, and the Shannon entropy of its probability distribution measures that uncertainty in bits. The exercise asks for this entropy for a fair coin and for a fair die, and how it behaves when the coin or the die is biased.

Coins, Dice and Entropy

A probability distribution $p$ on $n$ outcomes is a list of real numbers $p_1, \dots, p_n$ with every $p_i \ge 0$ and $\sum_i p_i = 1$. A coin has two outcomes, a die six.

The fair coin $u_2$ is the probability distribution on two outcomes that gives each outcome probability one half, $$u_2 = \bigl(\tfrac12, \tfrac12\bigr).$$(1)^✦ The fair die $u_6$ is the probability distribution on six outcomes that gives each face probability one sixth, $$u_6 = \bigl(\tfrac16, \tfrac16, \tfrac16, \tfrac16, \tfrac16, \tfrac16\bigr).$$(2)^✦

The Shannon entropy of a distribution $p$ on $n$ outcomes is the real number $$H(p) = -\sum_{i=1}^{n} p_i \ln p_i ,$$(3)^✦ where a term with $p_i = 0$ counts as zero. With the natural logarithm it is measured in nats. Divided by $\ln 2$ it becomes $-\sum_i p_i \log_2 p_i$, the entropy in bits.

The Claim

What Is Checked

The claims are stated over the entropy (3) of distributions on two and on six outcomes, and over the two fair distributions, the coin (1) and the die (2). What is proved about these two is their entropy in bits, (4) and (5), and that each is the one distribution of largest entropy on its outcomes, (6) and (7).