逻辑回归与最大熵模型

机器学习

---

Logistic Regression Model

Logistic distribution

The probability distribution of random variable $X$ obeying Logistic Distribution is

$$F(x) = P(X \le x) = \frac{1}{1 + e^{-(x-\mu)/\lambda}}$$

And its probability density function is

$$f(x) = F'(x) = \frac{e^{-(x-\mu)/\lambda}}{\lambda(1+e^{-(x-\mu)/\lambda})^2}$$

Sigmoid Function is logistic distribution function with $\mu = 0$ and $\lambda = 1$:

$$\mathrm{Sig}(x) = \frac{1}{1 + e^{-x}}$$

Binomial logistic regression model

Binomial logistic regression model is following probability distribution:

$$P(Y=1 \mid x) = \frac{\exp{(w \centerdot x)}}{1 + \exp{(w \centerdot x)}}$$

$$P(Y=0 \mid x) = \frac{1}{1 + \exp{(w \centerdot x)}}$$

Estimate model parameter

Minimize loss function

$$L(w) = -\sum_{i=1}^{N} \left[ y_i(w \centerdot x_i) - \log{(1 + \exp{(w \centerdot x_i)})} \right]$$

Multi-nomial logistic regression

Convert $k$-nomial random variable into $k-1$ binomial random variable.

Maximum Entropy Model

Maximum entropy theorem

Define Entropy as

$$H(p) = H(X) = -\sum_{i=1}^n p_i \log{p_i},$$
where $X$ is a random variable with probability $p_i = P(X = x_i)$, with range $0 \le H(p) \le \log{n}$.The larger the uncertainly is, the larger entropy $H(X)$ will be.

Define Conditional Entropy as

$$H(Y \mid X) = \sum_{i=1}^n p_i H(Y \mid X = x_i) = -\sum_{x, y} p(x) \times p(y \mid x) \log{(y \mid x)}$$
where $X$ and $p$ are defined as above.