Google

To create training data for the Next Sentence Prediction (NSP) task, pairs of sentences (SentA and SentB) are sampled. Positive examples are generated by taking two consecutive sentences from a text corpus. Negative examples are created by pairing a sentence with another sentence randomly selected from the corpus. This process effectively transforms the NSP task into a binary classification problem.

Training Data Generation for Next Sentence Prediction

A common method for creating a training dataset to determine if two sentences are consecutive is to pair a sentence with a random sentence from a different document to create a 'negative' example. Evaluate a potential weakness of this approach. Specifically, what kind of subtle sentence relationships might the model fail to learn to distinguish?

Critique of Negative Sample Generation

A data scientist is preparing a dataset to train a model that must determine if two sentences are consecutive. They observe that their trained model performs poorly, often incorrectly classifying two sentences from the same paragraph as consecutive, even when they are not. Analyze the data generation process described in the case study below and explain the most likely reason for the model's poor performance.

Diagnosing a Flaw in Training Data Generation

A language model is being trained on a task to determine if one sentence is the direct follow-up to another. The training data consists of sentence pairs. 'Positive' pairs are two sentences that appear consecutively in a text. 'Negative' pairs are created by taking a sentence and pairing it with a random sentence from a different part of the text.

Consider the following short text:
1. The rocket launched at dawn.
2. It soared through the atmosphere.
3. The crowd watched in awe.
4. Mission contr

Learn Before

Related