The following is explanation regarding "Entropy".
You can access the source jupyter notebook by following link. https://github.com/hiroshiu12/probability_statistic/blob/master/What_is_Entropy.ipynb

What is "Entropy" ?¶

0. Self-information¶

Before getting into "Entropy", I have to wrap my head around "Self-information".
Definition of "Self-information" $h(x)$ is

$$h(x) = -log(p(x))$$

How come "logarithm" is used? As far as I know, there are two reasons.

1. Decreasing function on the closed interval [0,1] is required.
2. We want to do addition of "Self-information" . However, joint probability is calculated by product of probability. Hence　function which meet $h(p1 * p2) = h(p1) + h(p2)$ is required.

1. Entropy¶

In [1]:

import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline

"Entropy " can be said as expectation of Self-information.
To put it simply,"Entropy" describe unpredictability of event.
Let event be $X (X_{0},X_{1},X_{2},\cdots,X_{n})$, and let probability of X be $P (P_{0},P_{1},P_{2},\cdots,P_{n})$.
"Entropy" $H(x)$ can be expressed as following. $$ H(x) = -\sum_{i=1}^{n}P_{i}\log_{2}P_{i}$$

In [2]:

def entropy(prob_list):
    """
    parameter
    ---------------------
    prob_list : prob_list is expected to be numpy. 
    This represents probability, hence "prob_list.sum()" should be 1.
    """
    return (-prob_list * np.log2(prob_list)).sum()

2. Example of "Bernoulli Trail"¶

As an example, I'm gonna observe entropy in "Bernoulli trial". Probability mass function of "Bernoulli trial" is described as bellow.

$$ Ber(x|\mu) = \mu^{x}(1-\mu)^{1-x}$$

Now "Bernoulli Trial" can be depicted as following function.

In [3]:

def bernoulli_trial(u):
    np_prob = np.empty((2,))
    np_prob[0] = (u ** 1)
    np_prob[1] = (1-u)
    
    return np_prob

Calculate probability of "Bernoulli Distribution"

In [4]:

prob_ber_list = np.arange(0.01,1,0.01)
ber_result_list = np.empty((prob_ber_list.shape[0],2))

# calculate probability of bernoulli
for i, prob_ber in enumerate(prob_ber_list):
    ber_result_list[i] = bernoulli_trial(prob_ber)
    
# calculate entropy
entropy_list = np.empty((prob_ber_list.shape[0],),dtype=np.float64)
for j, ber_result in enumerate(ber_result_list): 
    tmp = entropy(ber_result)
    entropy_list[j] = tmp

Entropy of "Bernoulli Distribution" can be depicted as following graph.

In [5]:

plt.plot(prob_ber_list,entropy_list)
plt.xlabel('Probability')
plt.ylabel('Entropy')
plt.title('Entropy of bernoulli distribution')

Out[5]:

<matplotlib.text.Text at 0x117fb9240>

As your intuition, when probability mass function accords to uniform distribution, Entropy $H(x)$ become maximum:)