Google

When Rotary Positional Embeddings (RoPE) are applied to query and key vectors, the self-attention mechanism inherently captures relative positional context. Specifically, if the RoPE-encoded vectors `Ro(x, tθ)` and `Ro(y, sθ)` are treated as the query and key respectively, the self-attention operation implicitly models the relative positions. This occurs because the attention score is derived from their dot product, which is a function of the relative displacement, $$t-s$$.

Implicit Relative Position Modeling in Self-Attention with RoPE

Consider an information-processing mechanism where each input item is represented by a 'query' vector and a 'key' vector. Before these vectors are compared, each is rotated in a 2D plane. The angle of rotation for an item at position 'p' is directly proportional to 'p'. The comparison score between a query from position 't' and a key from position 's' is then calculated using the dot product of their respective rotated vectors. A key mathematical property of this operation is that the resulting dot product value is a function of the original, un-rotated vectors and the difference in their positions, (t-s).

Based on this description, explain why this mechanism is said to capture *relative* positional information rather than *absolute* positional information.

Analysis of a Positional Encoding Method

Based on the description of the positional encoding method in the case study, evaluate the engineer's observations. Is the observed performance pattern (poor at absolute position tasks, strong at relative position tasks) consistent with the properties of the described encoding method? Justify your reasoning.

Evaluating a Model's Performance Discrepancy

An attention mechanism incorporates positional information by applying a unique rotation to each query and key vector based on its absolute position in a sequence. The attention score between a query from position 't' and a key from position 's' is then computed. A key property of this rotation is that the dot product between the rotated query and key vectors is a function of the original vectors and the difference in their positions (t-s). Based on this information, what can be concluded about 

Learn Before

Related