Google

To illustrate the generator's role in replaced token detection, an original sequence is first corrupted by masking certain tokens, which are then replaced by the predictions of a small masked language model (the generator). For instance:

$$
\begin{array}{r c c c c c c c c c}
\text{original:} & [\mathrm{CLS}] & \text{The} & \text{boy} & \text{spent} & \text{hours} & \text{working} & \text{on} & \text{toys} & . \\
& \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow \\
\text{masked:} & [\mathrm{CLS}] & \text{The} & \text{boy} & \text{spent} & [\mathrm{MASK}] & \text{working} & \text{on} & [\mathrm{MASK}] & . \\
& \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow \\
& \multicolumn{9}{c}{\text{Generator (small masked language model)}} \\
& \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow \\
\text{replaced:} & [\mathrm{CLS}] & \text{The} & \text{boy} & \text{spent} & \text{decades} & \text{working} & \text{on} & \text{toys} & .
\end{array}
$$

In this example, the masked token "hours" is incorrectly predicted and replaced by "decades", whereas "toys" is correctly predicted and replaced by the original token.

Visual Example of Generator Operation in Replaced Token Detection

A text-corrupting model is given the masked sentence: 'The chef prepared the [MASK] dish with [MASK] ingredients.' The original, unmasked sentence was: 'The chef prepared the signature dish with fresh ingredients.' The model outputs the following new sentence: 'The chef prepared the delicious dish with local ingredients.' Based on the function of this type of model, explain why the output tokens ('delicious', 'local') can be different from the original tokens ('signature', 'fresh').

Analyzing Generator Output in Text Corruption

The generator's role in replaced token detection involves several steps to create a plausible but altered version of an input sentence. Arrange the following events into the correct chronological sequence.

A small masked language model, acting as a generator, is given an input sequence where some tokens have been replaced with a [MASK] token: 'The quick [MASK] fox jumps over the [MASK] dog.' The original, unmasked sentence was: 'The quick brown fox jumps over the lazy dog.' Based on the typical behavior of such a generator, which of the following is the most plausible output sequence it would produce to be passed on for further processing?

Learn Before

Related