Google

This example illustrates the input and output for an encoder-decoder model undergoing denoising training. The encoder processes a corrupted input sequence where specific tokens have been replaced by a mask token, represented as: `[CLS] The puppies are [MASK] outside [MASK] house .` The decoder then utilizes the hidden representation from the encoder to predict and reconstruct the original, uncorrupted sequence, yielding: `⟨s⟩ The puppies are frolicking outside the house .`

Example of a Denoising Autoencoder Task for Encoder-Decoder Models

A researcher is training a model to reconstruct original sentences from corrupted versions where some words are replaced by a `[MASK]` token. The training process only calculates an error signal based on the model's predictions for these `[MASK]` positions. The researcher observes that the model becomes very good at filling in the blanks, but struggles to generate complete, fluent sentences from scratch. Explain why this specific training method might lead to this outcome.

Evaluating a Model's Training Objective

An encoder-decoder model is being trained on a denoising task. It receives the corrupted input `The [M] sat on the [M].` and is trained to reconstruct the original sentence, `The cat sat on the mat.`. During the training step for this example, on which specific words will the model's performance be evaluated to calculate the training loss? Explain why the evaluation is focused only on these words.

Focus of the Denoising Loss Function

A model is being trained to reconstruct an original text sequence from a corrupted version where some words have been replaced. If the model is given the corrupted input: `The quick brown [MASK] jumps over the [MASK] dog.`, which of the following would be the most appropriate target output for this training example?

Learn Before

Related