Entropy

In information theory, the entropy of a random variable is the average level of “information”. The core idea of information theory is that the “informational value” of a communicated message depends on the degree to which the content of the message is surprising. If a highly likely event occurs, the message carries very little information. On the other hand, if a highly unlikely event occurs, the message is much more informative. For instance, the knowledge that some particular number will not be the winning number of a lottery provides very little information, because any particular chosen number will almost certainly not win. However, knowledge that a particular number will win a lottery has high informational value because it communicates the outcome of a very low probability event.

Statements

Definition. Let $(\Omega ,\mathcal{F},P)$ be a probability space and $X:\Omega \to \mathcal{X}$ a random variable. The entropy of $H$ of $X$ is defined by the formula

$$H(X)=E[-\log p_X]=\{\begin{array}{llc}-\sum _{x\in \mathcal{X}}{p}_X(x)\log (p_X(x)),& \text{if}& X\text{is discrete}\\ -{\int }_{\mathcal{X}}{p}_X(x)\log (p_X(x)),& \text{if}& X\text{is continuous},\end{array}$$

where $p_x:\mathcal{X}\to [0,1]$ is the

• probability mass function, when $X$ is discrete
• probability density function, when $X$ is continuous. In this case, the entropy is also called as differential entropy.