https://towardsdatascience.com/softmax-function-simplified-714068bf8156

University of Michigan - Ann Arbor

Softmax regression is the generalization of logistic regression to multiple classes. In other words, each data point belongs to one of multiple classes (rather than just two options, as is the case for logistic regression). Hence softmax regression is also called multi-class logistic regression or multinomial logistic regression.

Softmax Regression (Activation)

https://www.kdnuggets.com/2016/07/softmax-regression-related-logistic-regression.html

What is Softmax Regression and How is it Related to Logistic Regression?

The output layer of softmax regression generates a probability distribution $$\hat{\mathbf{y}}$$, where each entry $$\hat{y}_i$$ represents the predicted probability that the input belongs to a particular class. These probabilities are calculated by applying the softmax activation function to the raw output scores $$\mathbf{o}$$. To ensure the outputs represent valid probabilities, the operation exponentiates each score and normalizes it by the sum of all exponentiated scores:

$$\hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}$$

This transforms the unconstrained linear model outputs into a valid probability distribution.

Output Layer of Softmax Regression

```
import numpy as np
def softmax_numpy(scores):
  return np.exp(scores)/sum(np.exp(scores), axis=0)
```

Implementation of Softmax Regression Using Numpy

```
import tensorflow as tf
def softmax_tensorflow(scores):
  return tf.exp(scores)/tf.reducesum(tf.exp(scores), 1) 
```

Implementation of Softmax Regression Using Tensorflow

For a pair of a one-hot label vector $$\mathbf{y}$$ and a model's predicted probability distribution $$\hat{\mathbf{y}}$$ over $$q$$ classes, the cross-entropy loss function is defined as:

$$l(\mathbf{y}, \hat{\mathbf{y}}) = - \sum_{j=1}^q y_j \log \hat{y}_j$$

Because $$\mathbf{y}$$ is a one-hot vector, the sum vanishes for all but the coordinate corresponding to the true class. This loss is bounded below by $$0$$ (since probabilities cannot exceed $$1$$ and their negative logarithm cannot be lower than $$0$$), and it only equals $$0$$ if the model predicts the true label with absolute certainty. However, reaching a probability of exactly $$1$$ requires infinite logits, so the loss is never completely $$0$$ for finite weights. Conversely, assigning an output probability of $$0$$ to the true label would incur an infinite loss ($$-\log 0 = \infty$$).

Cross-Entropy Loss for Softmax Regression

To maximize computational efficiency, the forward pass of a softmax regression model is typically vectorized across minibatches. For a minibatch of inputs $$\mathbf{X} \in \mathbb{R}^{n 	imes d}$$ containing $$n$$ examples with $$d$$ features, and parameters $$\mathbf{W} \in \mathbb{R}^{d 	imes q}$$ (weights) and $$\mathbf{b} \in \mathbb{R}^{1 	imes q}$$ (biases), the unnormalized logits are computed using the affine transformation $$\mathbf{O} = \mathbf{X} \mathbf{W} + \mathbf{b}$$. The softmax function is then applied rowwise to $$\mathbf{O}$$ to yield the normalized class probabilities $$\hat{\mathbf{Y}} = \mathrm{softmax}(\mathbf{O})$$ for the entire batch simultaneously.

Learn Before

Related