Processing math: 100%

Sunday, May 20, 2018

Fundamental understanding of "Logistic Regression"

Fundamental understanding of "Logistic Regression".

I've just looked back on "Logistic Regression". This is somewhat of memo for the time being:)

1. Basic logic behind the scene of "logistic regression".

In this article I will deal with only binary classification which is C1, and C2.
First of all, let us think about p(C1|x),
p(C1|x)=p(x|C1)p(C1)p(x)=p(x|C1)p(C1)p(x,C1)+p(x,C2)=p(x|C1)p(C1)p(x,C1)+p(x,C2)=p(x|C1)p(C1)p(x|C1)p(C1)+p(x|C2)p(C2)


Following that, let a be
a=logp(x|C1)p(C1)p(x|C2)p(C2)

We call p(x|C1)p(C1)p(x|C2)p(C2)=p(C1|x)p(x)p(C2|x)p(x)=p(C1|x)p(C2|x) as "odds". Logarithm of "Odds" is called as "log odds". Now, following is mathmaticaly trivial. ea=p(C2|x)p(x)p(C1|x)p(x)

Therefore, p(C1|x) can be expressed as bellow p(C1|x)=11+ea=σ(a)

We can call the function σ(a) as "Logistic Sigmoid Function". The "Inverse function" of "Logistic Sigmoid Function" is called "logit function".
a=logσ(a)1σ(a)=logp(C1|x)p(C2|x)
Let w be (w0,w1), x be (1,x), "Logistic regression" depict probability of p(1|x) as
p(1|x)=11+exp(w0+w1x)

Now let us assume a is w0+w1x, needless to say, this is "logistic sigmoid function". 

Note : This function is "non-linear function". In essense, <span style = "text-decoration:underline">we transform linear function a to non-linear funciton with "logistic sigmoid function" !</span>
So, Let us think about "odds" with w and x.
a=wTx=logp(C1|x)p(C2|x)


Hence,"odds" is expressed as, p(C1|x)p(C2|x)=ewTx

2. How the change with respect to x affect to odds??

Now we think about how the change in x affect to odds. Let us think about ˜x=(1,x+1)T, "odds ratio" would be,
p(C1|˜x)p(C2|˜x)p(C1|x)p(C1|x)=exp(wT˜x)exp(wTx)=exp(w1)


Consequently, addition of 1 in x will change odds rate of exp(w1).

No comments:

Post a Comment