Google

A common method for preparing data for sequence models involves concatenating input and output sequences into a single string, using a special separator token like `[SEP]` to distinguish between them. For instance, an input sequence `xm` and an output sequence `y1 y2 ... yn` can be formatted as `xm [SEP] y1 y2 ... yn [SEP]`. This structure allows the model to process both sequences simultaneously while understanding their distinct roles.

Using [SEP] Tokens for Sequence Concatenation

A data scientist is training a language model for a text summarization task. They have combined the original long text and its corresponding summary into a single sequence for each training example. However, the model is struggling to learn the task and is generating incoherent outputs. Below is an example of how one data point was formatted:

`'The Industrial Revolution was the transition to new manufacturing processes... [full long text] ... The Industrial Revolution transformed economies.'`

Based on common practices for preparing sequence data, identify the likely error in this formatting and explain why it would cause the model to perform poorly.

Diagnosing Model Training Issues from Data Formatting

Based on the provided case study, analyze the likely cause of the model's poor performance. Explain how the data formatting could be improved using a special separator token and why this change would help the model better distinguish between the article and its intended summary.

Debugging Data Preprocessing for a Summarization Model

When preparing parallel text for sequence models, such as translation pairs, the source and target sentences can be packed into a single concatenated sequence using special tokens. For example, a Chinese sentence "鲸鱼 是 哺乳 动物 。" and its English translation "Whales are mammals ." can be packed as: `[CLS] 鲸鱼 是 哺乳 动物 。 [SEP] Whales are mammals . [SEP]`. In this structure, the `[CLS]` token marks the beginning of the sequence, while `[SEP]` tokens are used to separate the two languages and indicate the end of the entire sequence.

Example of Sequence Packing for Translation

A language model is being prepared for a question-answering task. The model must process both the question and its corresponding answer as a single, combined sequence. If the question is 'What is the capital of France?' and the answer is 'Paris', how should these two sequences be formatted for the model using a special separator token to distinguish between them?

Learn Before

Related