PReLU

论文笔记

---

Kaiming He 在 Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classifcation 提出 PReLU，一种 fitting model 的激活函数，少额外计算，几乎不增加 overfitting 风险；以及一种对应的强健的初始化方法。

PReLU

$$f(y_i) = \begin{cases} y_i & y_i > 0 \\ a_i y_i & y_i \le 0 \end{cases}$$

MSRA weight initialization

Initialize weights with zero mean gaussian distribution whose std is

$$\frac 1 {n_l}$$
where $n_l = k_l^2 d_{l-1}$, $k_l$ is the spatial filter size in layer $l$ and $d_{l−1}$ is the number of filters in layer $l−1$.