Google

The Rotary Positional Embedding (RoPE) transformation can be expressed as an element-wise operation on a d-dimensional vector $$\mathbf{x}$$. This formulation calculates the rotated vector, $$\mathrm{Ro}(\mathbf{x}, t\theta)$$, by summing two vectors derived from Hadamard products (element-wise multiplication). The first vector is the product of the input vector $$\mathbf{x}$$ and a cosine vector. The second is the product of a permuted input vector ($$[-x_2, x_1, \dots, -x_d, x_{d-1}]$$) and a sine vector. The complete formula is: $$ \mathrm{Ro}(\mathbf{x}, t\theta) = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_{d-1} \\ x_d \end{pmatrix} \odot \begin{pmatrix} \cos t\theta_1 \\ \cos t\theta_1 \\ \vdots \\ \cos t\theta_{d/2} \\ \cos t\theta_{d/2} \end{pmatrix} + \begin{pmatrix} -x_2 \\ x_1 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix} \odot \begin{pmatrix} \sin t\theta_1 \\ \sin t\theta_1 \\ \vdots \\ \sin t\theta_{d/2} \\ \sin t\theta_{d/2} \end{pmatrix} $$ where $$\odot$$ denotes the Hadamard product.

Element-wise Formula for RoPE Rotation

In the element-wise formulation of Rotary Positional Embeddings (RoPE), a permuted version of the input vector $$\mathbf{x}$$ is required. This vector is constructed by swapping each pair of adjacent elements $$(x_{2k-1}, x_{2k})$$ and negating the first element of the swapped pair, resulting in $$(-x_{2k}, x_{2k-1})$$. For a full d-dimensional vector $$\mathbf{x}$$, the permuted vector is given by:

$$ \begin{pmatrix} -x_2 \\ x_1 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix} $$

This vector is a key component in the calculation, where it is multiplied element-wise with the sine vector.

Permuted Input Vector in RoPE

A transformation is applied to a d-dimensional vector $\mathbf{x}$ at position $t$ using a set of frequencies $\theta$. The transformation is defined by the following element-wise formula:
$$ \mathrm{Trans}(\mathbf{x}, t\theta) = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_{d-1} \\ x_d \end{pmatrix} \odot \begin{pmatrix} \cos t\theta_1 \\ \cos t\theta_1 \\ \vdots \\ \cos t\theta_{d/2} \\ \cos t\theta_{d/2} \end{pmatrix} + \begin{pmatrix} -x_2 \\ x_1 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix} \odot

A transformation is applied to a d-dimensional vector $\mathbf{x}$ at position $t$ using a set of frequencies $\theta$. The full transformation is defined by the following element-wise formula:
$$ \mathrm{Trans}_{full}(\mathbf{x}, t\theta) = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_{d-1} \\ x_d \end{pmatrix} \odot \begin{pmatrix} \cos t\theta_1 \\ \cos t\theta_1 \\ \vdots \\ \cos t\theta_{d/2} \\ \cos t\theta_{d/2} \end{pmatrix} + \begin{pmatrix} -x_2 \\ x_1 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix} \odot \begin{pmatrix} \sin t\theta_1 \\ \sin t\theta_1 \\ \vdots \\ \sin t\theta_{d/2} \\ \sin t\theta_{d/2} \end{pmatrix} $$
Now, consider a simplified version of this transformation where the second term (involving the sine vector) is removed:
$$ \mathrm{Trans}_{simple}(\mathbf{x}, t\theta) = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_{d-1} \\ x_d \end{pmatrix} \odot \begin{pmatrix} \cos t\theta_1 \\ \cos t\theta_1 \\ \vdots \\ \cos t\theta_{d/2} \\ \cos t\theta_{d/2} \end{pmatrix} $$
Analyze the effect of this simplification. What geometric property of the full transformation is lost in the simple version, and what operation does the simple version perform instead?

Analyzing Components of a Vector Transformation

Consider the element-wise formula for applying a rotational transformation to a d-dimensional vector $\mathbf{x}$. The formula involves summing two vectors derived from element-wise products. 

True or False: The first vector in the sum is the product of the original input vector $\mathbf{x}$ and a sine vector, and the second vector is the product of a permuted input vector (e.g., $[-x_2, x_1, \dots]$) and a cosine vector.

Learn Before

Related