Google

In BERT's Masked Language Modeling strategy, 10% of the tokens that are chosen for prediction are kept in their original, unchanged form within the input sequence.

Unchanged Tokens in BERT's MLM Strategy

To illustrate the strategy of leaving a selected token unchanged in BERT's Masked Language Modeling, consider the original input: `[CLS] It is raining . [SEP] I need an umbrella . [SEP]`. If the token 'I' is chosen for prediction but falls under the 10% rule where the token is left as is, the input sequence fed to the model remains identical to the original. Despite the token not being masked or altered, the model is still tasked with predicting 'I' based on the surrounding context.

Example of an Unchanged Token in a BERT Input Sequence

During a language model's pre-training phase, 15% of tokens in each sequence are selected for a prediction task. Of these selected tokens, 10% are left in their original form. If a given input sequence contains 4,000 tokens, how many tokens would you expect to be selected for prediction but remain unchanged in the input? Provide only the final numerical answer.

Calculating Token Modifications in Pre-training

A data scientist is pre-training a language model using a masked prediction objective. In their setup, 15% of the input tokens are selected for prediction. However, to simplify the process, they decide to replace 100% of these selected tokens with a special `[MASK]` token. After pre-training, they observe that the model's performance on various downstream fine-tuning tasks is significantly worse than expected. Evaluate the data scientist's simplified pre-training strategy. Why did this modification likely lead to poor performance during fine-tuning, despite the model performing well on the pre-training objective itself?

Critique of a Modified Pre-training Strategy

In BERT's Masked Language Modeling strategy, predicting a target token that has been intentionally left unchanged in the input sequence is a relatively simple task. The purpose of this strategy is to guide the model to utilize easier, more direct evidence for its predictions, as the original token is explicitly available in the provided context.

Purpose of Unchanged Tokens in BERT's MLM Strategy

A language model is pre-trained using a method where 15% of the words in an input sentence are selected for prediction. Of these selected words, a small fraction (10%) are intentionally left in their original form, while the model is still tasked with predicting them based on the surrounding context. What is the most significant reason for this strategy of leaving some target words unchanged?

Learn Before

Related