Entropy

The information carried by an event $\mathbf{x}$ can be quantified as:

$$I(\mathbf{x})=-\log p(\mathbf{x})$$

This means that the less the event is likely to happen (low $p(x)$) the higher the information it carries (high $\log p(x)$). In fact if $p(x)$ is high, its $\log$ will be near $0$, since a very probable event doesn’t carry much information.

Given a probability distribution $p$ on some set of events $x$, we define its entropy as the probability-weighted average of information:

H(p)= -\sum_\boldsymbol{x}p(\boldsymbol{x})\log p(\boldsymbol{x})