Google

For classification problems like Next Sentence Prediction (NSP), the loss function under maximum likelihood training is defined as the negative log-probability of the correct label given the sequence representation. The specific formula is:

$${}\mathrm{Loss}_{\mathrm{NSP}} = -\log \Pr(c_{\mathrm{gold}}|\mathbf{h}_{\mathrm{cls}})$$

where $${}c_{\mathrm{gold}}$$ represents the correct (or 'gold') label for the current sample, and $${}\mathbf{h}_{\mathrm{cls}}$$ is the aggregate sequence representation vector.

Next Sentence Prediction Loss Formula

A language model is being trained on a binary classification task to determine if sentence B is the actual sentence that follows sentence A. Consider two different training examples and the model's predictions for the correct label in each case. Based on the standard negative log-likelihood loss function used for such tasks, which example would result in a higher loss value, and why?

Analyzing Model Training Loss

For the task of predicting if two sentences are consecutive, a higher model-predicted probability for the correct label (e.g., 'IsNext' or 'NotNext') will result in a higher calculated loss value for that training example.

A language model is being trained on a task to determine if two sentences are consecutive. For a specific pair of sentences where the second sentence *is* the correct follow-up, the model's final classifier outputs a probability of 0.8 for the 'IsNext' label. Based on the standard negative log-likelihood loss function used for this task, what is the calculated loss value for this single training example? (Note: Use the natural logarithm, ln).

Learn Before

Related