Google

In a BERT-based encoder-decoder architecture, specific mathematical notation is used to represent the sequences involved in the process. The source text is denoted as a sequence of tokens, $$x_1...x_m$$, and its corresponding sequence of embeddings is represented as $$\mathbf{e}_{1}^{x}...\mathbf{e}_{m}^{x}$$. On the generation side, the target text sequence generated by the decoder is denoted as $$y_1...y_n$$, with its corresponding embedding sequence represented as $$\mathbf{e}_{1}^{y}...\mathbf{e}_{n}^{y}$$.

Notation in a BERT-based Encoder-Decoder Architecture

In a sequence-to-sequence model that uses a pre-trained text encoder, specific symbols are used to represent different parts of the data flow. Match each notational symbol with its correct description.

A sequence-to-sequence model is processing the source sentence 'Hello world', which is tokenized into two tokens. Using the standard notation where `ex` represents an embedding for a source token, write out the complete sequence of embeddings that is passed to the encoder. Remember to account for any special tokens that frame the input.

Encoder Input Notation

A sequence-to-sequence model is translating the source text 'The cat sat'. The model has already generated the first target token, 'Le'. To generate the *second* target token, what sequence of embeddings is fed into the decoder? Assume the decoder's input starts with a special token `<s>`, the source text embeddings are denoted by `ex`, and the target text embeddings are denoted by `ey`.

Learn Before

Related