Google

This example illustrates a masked language modeling task using the sentence `[C] The kitten is chasing the ball .` as the original text. The model is provided with an input where some words are replaced by a mask token, such as `[C] The kitten [M] [M] [M] ball .`. In this structure, `[C]` denotes the context and `[M]` represents the masked tokens. The model's objective is to predict the original words that were masked, which are 'is', 'chasing', and 'the'.

Example of Masked Language Modeling: Kitten Chasing Ball

A researcher wants to train a language model to specifically learn the relationship between a subject (who performs an action) and the verb (the action itself). Based on the two training strategies presented below, which one is more suitable for this goal? Justify your evaluation.

Evaluating Masking Strategies for Model Training

In a masked language modeling task with the input `The student opened their [M] to study for the exam.`, the model's prediction for the masked word is solely determined by the words that come before the mask (`The student opened their`).

A language model is being trained using a masked language modeling objective. If the original input sentence is 'The innovative chef prepares a delicious meal for the guests.', which of the following masked versions would be the most difficult for the model to reconstruct accurately?

Learn Before

Related