Create Histogram¶

"Histogram" is useful when I check dataset, more specifically, relation between explanatory variable and response variable.
Hence This is somewhat of memo of the way to create "Histogram"
Here, famaous and popular dataset "iris" is gonna be used.

In [1]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline
from sklearn.datasets import load_iris

First of all, check what the data is like.¶

In [2]:

iris_dataset = load_iris()

iris_data = iris_dataset.data
iris_target = iris_dataset.target

In [3]:

iris_data.shape

Out[3]:

(150, 4)

In [4]:

iris_data[0:5]

Out[4]:

array([[ 5.1,  3.5,  1.4,  0.2],
       [ 4.9,  3. ,  1.4,  0.2],
       [ 4.7,  3.2,  1.3,  0.2],
       [ 4.6,  3.1,  1.5,  0.2],
       [ 5. ,  3.6,  1.4,  0.2]])

In [5]:

# Check the variation of target data
np.unique(iris_target)

Out[5]:

array([0, 1, 2])

In [6]:

iris_target.shape

Out[6]:

(150,)

In [7]:

iris_target[0:20]

Out[7]:

array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

Create histogram¶

In [8]:

# create bins for first element of list
var_0 = iris_data[:,0]
count,bins = np.histogram(var_0,bins=30)

In [9]:

# seperate explanatory variable into Setosa, Versicoiour  and Virginica
setosa = iris_data[iris_target ==0]
versicoiour = iris_data[iris_target ==1]
virginica = iris_data[iris_target ==2]

In [10]:

plt.hist(setosa[:,0],bins=bins,alpha=0.5)
plt.hist(versicoiour[:,0],bins=bins,alpha=0.5)
plt.hist(virginica[:,0],bins=bins,alpha=0.5)

Out[10]:

(array([ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  1.,  1.,  3.,
         1.,  2.,  4.,  6.,  5.,  4.,  0.,  7.,  3.,  0.,  1.,  4.,  1.,
         0.,  1.,  4.,  1.]),
 array([ 4.3 ,  4.42,  4.54,  4.66,  4.78,  4.9 ,  5.02,  5.14,  5.26,
         5.38,  5.5 ,  5.62,  5.74,  5.86,  5.98,  6.1 ,  6.22,  6.34,
         6.46,  6.58,  6.7 ,  6.82,  6.94,  7.06,  7.18,  7.3 ,  7.42,
         7.54,  7.66,  7.78,  7.9 ]),
 <a list of 30 Patch objects>)

Consequently, we can see it seems to be hard to discern the species.
Now I'd like to observe all variables.

In [11]:

fig,axes = plt.subplots(2,2,figsize=(12,12))
axes_1dim = axes.ravel()

for i in range(iris_data.shape[1]):
    count,bins = np.histogram(iris_data[:,i],bins=30)
    axes_1dim[i].hist(setosa[:,i],bins=bins,alpha=0.5)
    axes_1dim[i].hist(versicoiour[:,i],bins=bins,alpha=0.5)
    axes_1dim[i].hist(virginica[:,i],bins=bins,alpha=0.5)

According to result above, it seems setosa can be discerned by only third of fourth explanatory variables:)

How to use gensim to obtain "bag-of-words" representation??

This is somewhat of memo to obtain "bag-of-words" representation by using gensim library.
You can access original code with following link. https://github.com/hiroshiu12/probability_statistic/blob/master/How_to_use_gensim.ipynb"

How to use gensim to obtain "bag-of-words" representation??¶

This notebook is just practice of https://radimrehurek.com/gensim/tut1.html. And reminder for me :)

In [36]:

from gensim import corpora
from gensim import matutils
import re

In [15]:

corpus = ["Membership of the club has dwindled from 70 to 20",
         "They tried to buffer themselves against problems and unvertainties",
         "I don't want to be just a cog in the wheel anymore"]

1. First of all, you have to tokenize corpus.¶

In [16]:

def simple_toknizer(corpus):
    """
    Parameter :
    -------------------
    corpus :  corpus should be corpus, list of sentences.
    """
    token_list = []
    for sentence in corpus:
        token_list.append(sentence.split(' '))
        
    return token_list

In [17]:

token_list = simple_toknizer(corpus)
print(token_list)

[['Membership', 'of', 'the', 'club', 'has', 'dwindled', 'from', '70', 'to', '20'], 
['They', 'tried', 'to', 'buffer', 'themselves', 'against', 'problems', 'and', 'unvertainties'],
 ['I', "don't", 'want', 'to', 'be', 'just', 'a', 'cog', 'in', 'the', 'wheel', 'anymore']]

2. Next you have to create dictionary.¶

"dictionary" is mapping between token and ids.

In [18]:

dictionary = corpora.Dictionary(token_list)
# You can check the mapping by caling 'token2id' attribute.
dictionary.token2id

Out[18]:

{'20': 9,
 '70': 7,
 'I': 18,
 'Membership': 0,
 'They': 10,
 'a': 23,
 'against': 14,
 'and': 16,
 'anymore': 27,
 'be': 21,
 'buffer': 12,
 'club': 3,
 'cog': 24,
 "don't": 19,
 'dwindled': 5,
 'from': 6,
 'has': 4,
 'in': 25,
 'just': 22,
 'of': 1,
 'problems': 15,
 'the': 2,
 'themselves': 13,
 'to': 8,
 'tried': 11,
 'unvertainties': 17,
 'want': 20,
 'wheel': 26}

Note :
You can mechanically filter some words out with 'filter_extremes' and 'fileter_n_most_frequent' methods.
Or you can specificly filter some words out with 'filter_tokens'.

As an example, filter numeric words from dictionary. They sometimes disrupt the model of machine learning or cluster..

In [26]:

# You must know the ids of word you want omit.
regular_exp = re.compile('\d+')
ids_list = []
for word, number in dictionary.token2id.items():
    if regular_exp.match(word):
        ids_list.append(number)

print('The number you wanna filter out is : ',ids_list)

The number you wanna filter out is :  [7, 9]

Caution ! :¶

1. "filter_tokens" method modify mapping of itself.
2. It changes ids of dictionary.

In [27]:

dictionary.filter_tokens(bad_ids=ids_list)

In [29]:

# ids were changed.
dictionary.token2id

Out[29]:

{'I': 16,
 'Membership': 0,
 'They': 8,
 'a': 21,
 'against': 12,
 'and': 14,
 'anymore': 25,
 'be': 19,
 'buffer': 10,
 'club': 3,
 'cog': 22,
 "don't": 17,
 'dwindled': 5,
 'from': 6,
 'has': 4,
 'in': 23,
 'just': 20,
 'of': 1,
 'problems': 13,
 'the': 2,
 'themselves': 11,
 'to': 7,
 'tried': 9,
 'unvertainties': 15,
 'want': 18,
 'wheel': 24}

3. Now is the time to create bag of words representation with sparse vector.¶

Contents of sparse vector is tuple which is (word id, the number of occurrence)

In [35]:

sparse_vector = [dictionary.doc2bow(tokens) for tokens in token_list] 
sparse_vector

Out[35]:

[[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)],
 [(7, 1),
  (8, 1),
  (9, 1),
  (10, 1),
  (11, 1),
  (12, 1),
  (13, 1),
  (14, 1),
  (15, 1)],
 [(2, 1),
  (7, 1),
  (16, 1),
  (17, 1),
  (18, 1),
  (19, 1),
  (20, 1),
  (21, 1),
  (22, 1),
  (23, 1),
  (24, 1),
  (25, 1)]]

4. gensim has useful uitility to make dense vector.¶

You can simply use matutils.corpus2dense to obtain dense vector, however don't forget let it know the number of dimention. Since dimentionality cannot be deduced from sparse vector.

In [40]:

dense_vector= matutils.corpus2dense(sparse_vector,num_terms=len(dictionary.token2id))
dense_vector

Out[40]:

array([[ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  1.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  1.,  1.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.]], dtype=float32)

However, mostly we use transposed version of that dense vector for machine learning or cluster or etc...
Therefore transposed vector is more useful :)

In [42]:

dense_vector.T

Out[42]:

array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,
         0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,  1.,
         1.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,
         0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]], dtype=float32)

The following is explanation regarding "Entropy".
You can access the source jupyter notebook by following link. https://github.com/hiroshiu12/probability_statistic/blob/master/What_is_Entropy.ipynb

What is "Entropy" ?¶

0. Self-information¶

Before getting into "Entropy", I have to wrap my head around "Self-information".
Definition of "Self-information" $h(x)$ is

$$h(x) = -log(p(x))$$

How come "logarithm" is used? As far as I know, there are two reasons.

1. Decreasing function on the closed interval [0,1] is required.
2. We want to do addition of "Self-information" . However, joint probability is calculated by product of probability. Hence　function which meet $h(p1 * p2) = h(p1) + h(p2)$ is required.

1. Entropy¶

In [1]:

import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline

"Entropy " can be said as expectation of Self-information.
To put it simply,"Entropy" describe unpredictability of event.
Let event be $X (X_{0},X_{1},X_{2},\cdots,X_{n})$, and let probability of X be $P (P_{0},P_{1},P_{2},\cdots,P_{n})$.
"Entropy" $H(x)$ can be expressed as following. $$ H(x) = -\sum_{i=1}^{n}P_{i}\log_{2}P_{i}$$

In [2]:

def entropy(prob_list):
    """
    parameter
    ---------------------
    prob_list : prob_list is expected to be numpy. 
    This represents probability, hence "prob_list.sum()" should be 1.
    """
    return (-prob_list * np.log2(prob_list)).sum()

2. Example of "Bernoulli Trail"¶

As an example, I'm gonna observe entropy in "Bernoulli trial". Probability mass function of "Bernoulli trial" is described as bellow.

$$ Ber(x|\mu) = \mu^{x}(1-\mu)^{1-x}$$

Now "Bernoulli Trial" can be depicted as following function.

In [3]:

def bernoulli_trial(u):
    np_prob = np.empty((2,))
    np_prob[0] = (u ** 1)
    np_prob[1] = (1-u)
    
    return np_prob

Calculate probability of "Bernoulli Distribution"

In [4]:

prob_ber_list = np.arange(0.01,1,0.01)
ber_result_list = np.empty((prob_ber_list.shape[0],2))

# calculate probability of bernoulli
for i, prob_ber in enumerate(prob_ber_list):
    ber_result_list[i] = bernoulli_trial(prob_ber)
    
# calculate entropy
entropy_list = np.empty((prob_ber_list.shape[0],),dtype=np.float64)
for j, ber_result in enumerate(ber_result_list): 
    tmp = entropy(ber_result)
    entropy_list[j] = tmp

Entropy of "Bernoulli Distribution" can be depicted as following graph.

In [5]:

plt.plot(prob_ber_list,entropy_list)
plt.xlabel('Probability')
plt.ylabel('Entropy')
plt.title('Entropy of bernoulli distribution')

Out[5]:

<matplotlib.text.Text at 0x117fb9240>

As your intuition, when probability mass function accords to uniform distribution, Entropy $H(x)$ become maximum:)

How to sort list or numpy in descending order in Python??

Unfortunately, there seems not to be function which sort list or numpy in descending order.Therefore through this article, I'd like to share the way to sort list or numpy in descending order:)

list

There is two ways to sort in descending order.

In [1]: test_list = [3,2,4,8]

In [2]: test_list.sort()

In [3]: test_list
Out[3]: [2, 3, 4, 8]

In [4]: test_list.reverse()

In [5]: test_list
Out[5]: [8, 4, 3, 2]



In [1]: test_list = [3,2,4,8]

In [2]: test_list.sort()

In [3]: test_list[::-1]
Out[3]: [8, 4, 3, 2]

In [4]: test_list
Out[4]: [2, 3, 4, 8]

In [5]: 

Notice ! : First one is modify list in-place, however, Second one (slice) is to return new sorted list, hence original list stays as it is .

Incidentally, In terms of in-place or not, built-in method which is called "sorted" returns new sorted list, while "list.sort()" modify list in-place. In order to avoid confusion, "list.sort()" is to return None. The following is sample.

In [1]: test_list = [3,2,4,8]

In [2]: sorted(test_list)
Out[2]: [2, 3, 4, 8]

In [3]: test_list
Out[3]: [3, 2, 4, 8]

In [4]: 

numpy

I usually do as bellow.

In [1]: import numpy as np

In [2]: test_np = np.array([3,2,4,8])

In [3]: np.sort(test_np)[::-1]
Out[3]: array([8, 4, 3, 2])

Incidentally, sometimes we need sorted index of numpy. In that case, useful　mehod "argsort" shoud be used as bellow.

In [4]: np.argsort(test_np)
Out[4]: array([1, 0, 2, 3])

In [5]: np.argsort(test_np)[::-1]
Out[5]: array([3, 2, 0, 1])

Binomial Distribution

The binomial distribution is one of discrete probability distribution　of the number of success in a sequence of statistically independent bernoulli trail.
I've just tried to created graph of binomial distribution with python. Hence it's somewhat of memo of that:)

Prepare the calss of bernoulli trial

At first, I created python class of bernoulli trial which return probability of bernoulli trial with parameter n,k,p. Following is what I implemented.

Create graph of binomial distribution with various probability of success

Following is the graph with various probability of success in bernoulli trial.

From the result bellow, the vertex of probability distribution seems to be determined by probability of success in bernoulli trial.

Create graph of binomial distribution with various the number of bernoulli traial

Following is the graph with various number of bernoulli trial.

How to use '@' option in zip command ??

You can specify files to be zipped from standard input by using '@' option as following.

    
$ touch {a,b,c,d}
$ ls
a b c d
$ ls -ltrh
total 0
-rw-r--r--  1 user1  staff     0B  2  1 16:44 d
-rw-r--r--  1 user1  staff     0B  2  1 16:44 c
-rw-r--r--  1 user1  staff     0B  2  1 16:44 b
-rw-r--r--  1 user1  staff     0B  2  1 16:44 a
$ vi filelist
$ 
$ cat ./filelist 
a
b
c
d
$ cat ./filelist | zip -@ ./test.zip
  adding: a (stored 0%)
  adding: b (stored 0%)
  adding: c (stored 0%)
  adding: d (stored 0%)
$ ls
a        b        c        d        filelist test.zip
$ mkdir result
$ unzip ./test.zip -d $_
Archive:  ./test.zip
 extracting: ./result/a              
 extracting: ./result/b              
 extracting: ./result/c              
 extracting: ./result/d              
$ ls ./result/
a b c d
$ 
    

In case that there are several files in a directory, and partially you wanna zip, you can specify which file you wanna zip thorough file-list or standard input.

Read file in Python

When opening file in python, I think, crucial thing is comprehending '__iter__()' and 'next()' is implemented in TextIOWrapper. Because of that, you can read contents of file as bellow. And that's the standard and best way to read all contents of file line by line.

    
        with open('file name') as f:
            for line in f:
                print(line)
    

Following is the old way to read file in python. Just for memo.
After reading all contents of file, 'readline()' method will return empty string as ''. And boolean value of '' is False. As such EOF can be handled.

    
        with open('file name') as f:
            while True:
                line = f.readline()
                if not line:
                    break
                print(line)