Google

An example of an input with two unrelated sentences for a Next Sentence Prediction (NSP) task is `[CLS] The cat sleeps on the windowsill . [SEP] Apples grow on trees . [SEP]`. Because the second sentence does not logically follow the first, this training sample is classified with the label `NotNext`. This serves as a negative example to train the classification system.

Example of an Unrelated Sentence Pair for NSP

A language model is being trained to perform a classification task to determine if two sentences are logically connected. Which of the following inputs would serve as the most effective *negative example* for this training process?

You are preparing training data for a language model on a task that involves determining if two sentences are consecutive. Construct a single input string that represents a *negative example* (where the second sentence does not logically follow the first). Your example must use two distinct, unrelated sentences and include the necessary special tokens in their correct positions.

Constructing a Negative Training Example

For a task where a model learns to predict if one sentence logically follows another, a negative training example is formatted as `[CLS] Sentence A . [SEP] Sentence B . [SEP]`. In this format, the `[SEP]` token's specific function is to explicitly label the sentence pair as unrelated.

Learn Before

Related