Google

The Masked Language Modeling (MLM) framework offers significant flexibility for training encoder-decoder models. Different training objectives can be created by adjusting various parameters, such as the percentage of tokens that are masked and the maximum length of the text spans that are replaced by a mask token. This adaptability allows the training to range from a BERT-style objective with partial masking to a full language modeling task where the entire sequence is generated.

Flexibility of Masked Language Modeling for Encoder-Decoder Training

An extreme application of Masked Language Modeling (MLM) involves masking 100% of the tokens in a sequence, which effectively transforms the training objective into a sequence generation task. In this scenario, an input consisting of a `[CLS]` token followed by `[MASK]` tokens for every word is used to train the model to generate the complete, original sentence. For example: `[CLS] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] → ⟨s⟩ The puppies are frolicking outside the house .`

Example of Full Sequence Generation via 100% Masking

A research team is developing a new encoder-decoder model for natural language understanding. Their goal is to create a single pre-trained model that can be effectively fine-tuned for two distinct tasks: (1) generating concise summaries of long articles, and (2) restoring missing phrases in damaged historical texts. They are considering two different pre-training objectives based on masking input text.

*   **Objective A:** Randomly replace 15% of the individual words in the input text with a special `[MASK]` token. The model must predict the original words at these masked positions.
*   **Objective B:** Replace contiguous spans of text, totaling approximately 35% of the input, with a single `[MASK]` token. The model must generate the entire missing text span.

Evaluate which of these two pre-training objectives is more suitable for achieving the team's dual goals. Justify your choice by explaining how each objective would likely influence the model's capabilities for the target tasks.

Evaluating Pre-training Objectives for a Multi-Task Model

Match each masked language modeling (MLM) pre-training strategy for an encoder-decoder model with the primary capability it is designed to develop.

A research team is pre-training two separate encoder-decoder models using different variations of a masked language modeling objective.

*   **Model A** is trained by masking 15% of the input tokens, with each mask covering only a single token. The model's objective is to predict the original token for each masked position.
*   **Model B** is trained by masking 50% of the input tokens, with masks covering contiguous spans of up to 10 tokens. The model's objective is to predict the entire origina

Learn Before

Related