CNNVR: Neural Networks, backpropagation

机器学习

---

Chain Rule

$$\frac{\partial f}{\partial x} = \frac{\partial f}{\partial g} \frac{\partial g}{\partial x}$$

NN

Neural network Output Axon:

$$a = f \left(\sum_i w_i x_i + b \right)$$

Sigmoid Function

$$\frac{\partial \sigma(x)}{\partial x} = \left[1 - \sigma(x) \right] \sigma(x)$$
for sigmoid function $\sigma(x) = \frac{1}{1 + \exp(-x)}$.

Using sigmoid as activation, 2 problems:

• Saturated neuron kill the gradients
• Sigmoid outputs are not zero-centered
  If input to a neuron is always positive, since $\frac{\partial \sigma(x)}{\partial x} \gt 0$ and $x_j \gt 0$, all the gradients will always be positive. That's not good.

TanH

• Squashes numbers to range $[-1, 1]$
• Zero centered (nice)
• Still kills gradients when saturated

ReLU

$$f(x) = \max(0, x)$$
Pros:

• Doesn't saturate
• Computaionally effective
• Converges much faster than sigmoid and tanh

But:
Never update if $x \lt 0$. Use Leaky ReLU instead.

Maxout

$$\max{(w_1 x + b_1, w_2 x + b_2)}$$

In Pratice

Use ReLU, and be careful with learning rate. Tryout Leaky ReLU and Maxout. Don't expect tanh too much. Never use sigmoid.