Google

To illustrate the strategy of leaving a selected token unchanged in BERT's Masked Language Modeling, consider the original input: `[CLS] It is raining . [SEP] I need an umbrella . [SEP]`. If the token 'I' is chosen for prediction but falls under the 10% rule where the token is left as is, the input sequence fed to the model remains identical to the original. Despite the token not being masked or altered, the model is still tasked with predicting 'I' based on the surrounding context.

Example of an Unchanged Token in a BERT Input Sequence

Based on the training instance described below, is the engineer's conclusion that there is an error correct? Justify your answer by explaining the training principle at play.

Analyzing a Language Model's Training Process

Consider the following input sequence for a language model: `[CLS] The sky is blue . [SEP]`. During a specific training step, the word 'is' is selected for prediction. However, due to a particular training rule, the input sequence given to the model remains exactly `[CLS] The sky is blue . [SEP]`. Describe what the model is tasked to do with the word 'is' in this specific scenario, despite it being visible in the input.

Language Model Training Task

During a language model's training, a specific token is chosen from an input sequence to be predicted. In a small percentage of cases, the training strategy requires this chosen token to be left as-is, without being replaced. Consider the original sequence: `[CLS] The quick brown fox jumps . [SEP]`. If the token 'fox' is selected for prediction but falls under the rule where it remains unchanged, what is the final input sequence fed to the model for this training step?

Learn Before

Related