统计学习方法概论

机器学习

---

Statistical learning

Also unknown as Statistical machine learning.

• supervised
• unsupervised
• semi-supervised

3 Keys in machine learning

• Model
• Stategy
• Algorithm

Model

Denotes hypothesis space as

$$\mathcal{F} = \{ f | Y = f_\theta(X), \theta \in \mathbf{R}^n \},$$
where $X$ is input variable in input space $\mathcal{X}$ and $Y$ is output variable in output space $\mathcal{Y}$, $\mathbf{R}^n$ vector is parameter from $n$ dimensional parameter space.

Similarily, we can define hypothesis space in probability view as

$$\mathcal{F} = \{P | P(Y|X)\}.$$

Strategy

Loss function

• 0-1 loss function
• euclidean loss function
• absolute loss function
• log likelihood loss functin

Definie expectation of loss as

$$R_{exp}(f) = \int_{x \times y} L(y, f(x)) P(x, y) dxdy.$$
But you actually need empirical loss:
$$R_{emp}(f) = \frac{1}{N} \sum_{i=1}^{N}L(y_i, f(x_i)).$$

ERM & SRM

ERM means empirical risk minimization (minimizing empirical loss function). But it causes over-fitting problem when trainning set is not large enough.

SRM (structural risk minimization) is a strategy proposed against over-fitting problem. Structural risk minimization is equivalent to regularization. SRM is defined as:

$$R_{srm}(f) = \frac{1}{N}\sum_{i=1}^{N}L(y_i, f(x_i)) + \lambda J(f),$$
where $J(f)$ is the complexity of the model. You target is to minimizing SRM.

Algorithm

The approach to find optimized model. I guess SGD is most popular in practice.

Model Evaluation and Selection

Too complicated model causes over-fitting problems.

Regularization and Cross validation

Regularization

Simply add regularizer (or penalty term):

$$L(w) = \frac{1}{N} \sum_{i=1}^{N} (f(x_i; w) - y_i)^2 + \frac{\lambda}{2} \|w\|^2,$$
L1 norm works as well:
$$L(w) = \frac{1}{N} \sum_{i=1}^{N} (f(x_i; w) - y_i)^2 + \frac{\lambda}{2} \|w\|_1.$$
Occam's razor: The simpler the model is, the better the model is.

Cross validation

S-fold cross validation is the most often used cross validation method. Split dataset into $s$ subsets, uses $s -1$ of them for training and $1$ for validation.

Generalization ability

Generalization error

Generalization error reflects the generalization ability of model learnt.

Generalization error bound