Softmax Classifier

机器学习

---

Hypothesis takes the form

$$h_\theta(x) = \left[ \begin{matrix} P(y=1 \mid x; \theta) \\ \vdots \\ P(y=K \mid x; \theta) \end{matrix} \right] = \frac 1 {\sum_1^K \exp(\theta^{(j)} x)} \left[ \begin{matrix} \exp(\theta^{(1)} x) \\ \vdots \\ \exp(\theta^{(K)} x) \end{matrix} \right]$$

The cost function will be

$$J(\theta) = - \left[ \sum_{i=1}^m \sum_{k=1}^K I(y^{(i)} = k) \log{ \frac{\exp(\theta^{(k)} x)}{ \sum_{j=1}^K \exp(\theta^{(j)} x) } } \right]$$

The gradient is

$$\nabla_{\theta^{(k)}} J(\theta) = -\sum_{i=1}^m \left[ x^{(i)} \left( I(y^{(i)} = k) - P(y^{(i)} = k \mid x^{(i)}; \theta) \right) \right]$$