Google

Positional Encoding, denoted as $$PE(i)$$, is a technique used to inject information about the relative or absolute position of tokens in a sequence. The encoding for a specific position $$i$$ is represented as a vector in a $$d$$-dimensional real number space. This is formally expressed as $$PE(i) \in \mathbb{R}^d$$, meaning each position is mapped to a unique high-dimensional vector.

Positional Encoding Vector

A language translation model is designed to process all words in a sentence at the same time, rather than one after another. When translating the English sentence "The quick brown fox jumps over the lazy dog," it produces the nonsensical German output "Fuchs springt über Hund der faule braune schnelle der." The model correctly translated each word, but the order is completely scrambled. What specific type of information, which must be represented for each word, is this model likely failing to consider? Explain your reasoning.

Diagnosing a Model's Translation Error

Given the following case study, identify the most likely flaw in the model's method for handling word order and explain your reasoning.

Analyzing a Model's Order-Insensitivity

A sequence processing model is configured to represent all its internal data using 512-dimensional vectors. To understand the order of items, it generates a unique vector for each position in an input sequence. If the model is given a sequence with 40 items, what are the dimensions of the vector that represents the position of the 25th item?

Learn Before

Related