Google

The Rotary Positional Embedding (RoPE) rotation is applied to a two-dimensional vector $$x = \begin{bmatrix} x_1 & x_2 \end{bmatrix}$$ by post-multiplying the vector by the corresponding 2D rotation matrix, $$R_\theta$$. The resulting rotated vector's components are calculated as follows: $$ \mathrm{Ro}(x, \theta) = xR_\theta = \begin{bmatrix} x_1 & x_2 \end{bmatrix} \begin{bmatrix} \cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{bmatrix} = \begin{bmatrix} \cos \theta \cdot x_1 - \sin \theta \cdot x_2 & \sin \theta \cdot x_1 + \cos \theta \cdot x_2 \end{bmatrix} $$

Application of RoPE Rotation to a 2D Vector

Consider the transformation applied to a two-dimensional vector $x = \begin{bmatrix} x_1 & x_2 \end{bmatrix}$ by an angle $\theta$, resulting in a new vector $y = \begin{bmatrix} \cos \theta \cdot x_1 - \sin \theta \cdot x_2 & \sin \theta \cdot x_1 + \cos \theta \cdot x_2 \end{bmatrix}$. This transformation will always alter the magnitude (or length) of the original vector $x$.

An AI developer is implementing a positional embedding layer. They need to apply a 2D rotation to vector components, but it is critical that the transformation does not change the vector's length (magnitude), as this would distort the information encoded in the embedding. They are considering two potential transformation functions for a vector $x = \begin{bmatrix} x_1 & x_2 \end{bmatrix}$ and an angle $\theta$:

- **Function A:** $y = \begin{bmatrix} \cos \theta \cdot x_1 - \sin \theta \cdot x_2 & \sin \theta \cdot x_1 + \cos \theta \cdot x_2 \end{bmatrix}$
- **Function B:** $y = \begin{bmatrix} \cos \theta \cdot x_1 + \sin \theta \cdot x_2 & \sin \theta \cdot x_1 + \cos \theta \cdot x_2 \end{bmatrix}$

Analyze both functions. Which function should the developer choose to ensure the vector's magnitude is preserved after the transformation? Justify your answer by explaining why the chosen function works and the other one fails.

Analysis of Rotational Transformation Properties

The dot product of two RoPE-encoded vectors in 2D Euclidean space, one at position $$t$$ and another at position $$s$$, can be simplified to demonstrate its dependency on the relative position $$(t-s)$$. The derivation uses the matrix representation of the rotation and two key properties: the transpose of a rotation matrix represents rotation by the negative angle ($$R_{s\theta}^\mathrm{T} = R_{-s\theta}$$), and multiplying rotation matrices is equivalent to adding their angles ($$R_{t\theta}R_{-s\theta} = R_{(t-s)\theta}$$). Thus:

$$ \mathrm{Ro}(\mathbf{x}, t\theta)[\mathrm{Ro}(\mathbf{y}, s\theta)]^\mathrm{T} = \mathbf{x}R_{t\theta}[\mathbf{y}R_{s\theta}]^\mathrm{T} = \mathbf{x}R_{t\theta}[R_{s\theta}]^\mathrm{T}\mathbf{y}^\mathrm{T} = \mathbf{x}R_{(t-s)\theta}\mathbf{y}^\mathrm{T} $$

This result shows that the dot product is equivalent to rotating vector $$\mathbf{x}$$ by the relative angle $$(t-s)\theta$$ and then taking its dot product with vector $$\mathbf{y}$$.

Derivation of the Dot Product for RoPE-Encoded Vectors

A two-dimensional vector is represented as `x = [3, 4]`. Calculate the resulting vector after applying a rotational transformation with an angle `θ = 90°` (π/2 radians).

Learn Before

Related