Google

To train a language model, such as a decoder-only architecture, the standard approach is to minimize a loss function over a collection of token sequences. This function, denoted as $$\mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}})$$, measures the discrepancy between the model's predicted probability distribution $$\mathbf{p}_{i+1}^{\theta}$$ and the true, gold-standard distribution $$\mathbf{p}_{i+1}^{\mathrm{gold}}$$ at each position. In natural language processing, this difference is typically quantified using the log-scale cross-entropy loss.

Loss Function for Language Modeling

The total loss for a given sequence of $$m$$ tokens $$(x_0, \dots, x_m)$$ is computed by summing the individual losses over each position from $$i=0$$ to $$m-1$$. At each position $$i$$, a loss function $$\mathcal{L}$$ measures the discrepancy between the model's predicted probability distribution for the next token ($$\mathbf{p}_{i+1}^{\theta}$$) and the actual ground-truth distribution ($$\mathbf{p}_{i+1}^{\mathrm{gold}}$$). This is expressed generally as:

$$ \mathrm{Loss}_{\theta}(x_0, \dots, x_m) = \sum_{i=0}^{m-1} \mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}}) $$

In natural language processing, this loss function $$\mathcal{L}$$ is typically the log-scale cross-entropy loss, leading to the specific formula:

$$ \mathrm{Loss}_{\theta}(x_0, \dots, x_m) = \sum_{i=0}^{m-1} \mathrm{LogCrossEntropy}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}}) $$

Total Loss Calculation for a Token Sequence

A language model is being trained to complete the sentence 'The sky is...'. The training data indicates the correct next word is 'blue'. The model's performance is measured by a value that quantifies the difference between its predicted probabilities and the correct outcome. A lower value indicates a better prediction.

Two different versions of the model produce the following probability distributions for the next word:

*   **Model Alpha:** Assigns a probability of 0.8 to 'blue', 0.1 to 'green', and 0.1 to 'cloudy'.
*   **Model Beta:** Assigns a probability of 0.3 to 'blue', 0.5 to 'cloudy', and 0.2 to 'green'.

Which model, Alpha or Beta, would be assigned a lower performance penalty (i.e., a lower loss value) for this specific prediction? Justify your reasoning.

Evaluating Model Prediction Quality

A language model with a vocabulary of ['mat', 'hat', 'sat', 'cat', 'the'] is predicting the next word for the context 'The cat sat on the'. The correct next word is 'mat'. To calculate the error for this prediction, a loss function compares the model's predicted probability distribution to a 'gold' or ground truth distribution. Describe the 'gold' distribution for this specific case, representing it as a vector.

Defining the Ground Truth Distribution

A language model is being trained to predict the next word in a sequence. The training process aims to minimize a loss value, which measures the difference between the model's predicted probability distribution for the next word and the actual correct word. Consider two separate predictions for the next word after the phrase 'The sun is shining...':

- **Prediction A:** The model assigns a probability of 0.75 to the correct word, 'brightly'.
- **Prediction B:** The model assigns a probability of

Learn Before

Related