Google

This diagram provides an example of training an encoder-decoder model using a denoising autoencoding objective. The process involves several key steps: 1. **Corrupted Input to Encoder:** The encoder receives a corrupted version of a sentence where some tokens are masked, for instance, `[CLS] The puppies are [MASK] outside [MASK] house .`. 2. **Sequence Reconstruction by Decoder:** The encoder generates a hidden state representation of the input, which is then passed to the decoder. The decoder's task is to reconstruct the original, uncorrupted sentence, `⟨s⟩ The puppies are frolicking outside the house .`, in an autoregressive manner. 3. **Sequence-Level Loss Calculation:** To train the model, a loss is calculated over the entire output sequence by accumulating the losses of all tokens, as in standard language modeling. This involves comparing the decoder's generated output with the ground-truth sequence, and the resulting error signal is used to update the model's parameters.

Diagrammatic Example of an Encoder-Decoder Model Trained with a Denoising Autoencoding Objective

Based on the provided scenario, explain how the training loss is calculated and what this loss signal is used for in the model's learning process.

Analyzing a Denoising Training Process

A researcher is training an encoder-decoder model using a denoising objective, where the model learns to reconstruct an original sentence from a corrupted version. Arrange the following steps of a single training iteration in the correct chronological order.

An encoder-decoder model is trained on a denoising task. It receives a corrupted input like `The quick [M] fox jumps [M] the lazy dog.` and must generate the original, complete sentence `The quick brown fox jumps over the lazy dog.` The decoder generates the output one word at a time. Why is the training loss typically calculated for each word the decoder generates, rather than just a single loss for the entire completed sentence?

Learn Before

Related