逻辑回归与最大熵模型
机器学习
Logistic Regression Model
Logistic distribution
The probability distribution of random variable X obeying Logistic Distribution is
F(x)=P(X≤x)=11+e−(x−μ)/λ
And its probability density function is
f(x)=F′(x)=e−(x−μ)/λλ(1+e−(x−μ)/λ)2
Sigmoid Function is logistic distribution function with μ=0 and λ=1:
Sig(x)=11+e−x
Binomial logistic regression model
Binomial logistic regression model is following probability distribution:
P(Y=1∣x)=exp(w⋅x)1+exp(w⋅x)
P(Y=0∣x)=11+exp(w⋅x)
Estimate model parameter
Minimize loss function
L(w)=−∑i=1N[yi(w⋅xi)−log(1+exp(w⋅xi))]
Multi-nomial logistic regression
Convert k-nomial random variable into k−1 binomial random variable.
Maximum Entropy Model
Maximum entropy theorem
Define Entropy as
H(p)=H(X)=−∑i=1npilogpi,
where
X is a random variable with probability
pi=P(X=xi), with range
0≤H(p)≤logn.The larger the uncertainly is, the larger entropy
H(X) will be.
Define Conditional Entropy as
H(Y∣X)=∑i=1npiH(Y∣X=xi)=−∑x,yp(x)×p(y∣x)log(y∣x)
where
X and
p are defined as above.