Google

While the formula for Rotary Positional Embeddings (RoPE) in complex space, $$C(\mathbf{x}, t\theta)$$, appears intricate, it has an equivalent and more intuitive formulation within the d-dimensional Euclidean space.

Comparison of Complex and Euclidean Representations of RoPE Rotation

The Euclidean space representation of Rotary Positional Embeddings (RoPE) applies a rotational transformation to a d-dimensional vector $$\mathbf{x}$$ through matrix multiplication. This transformation is performed by a d-dimensional square rotation matrix, denoted as $$R(t) \in \mathbb{R}^{d \times d}$$, which is specific to the token's position $$t$$. The formula is: $$ \mathrm{Ro}(\mathbf{x}, t\theta) = \mathbf{x} R(t) = \begin{bmatrix} x_1 & x_2 & \dots & x_d \end{bmatrix} \begin{bmatrix} R_{t\theta_1} & & & \\ & R_{t\theta_2} & & \\ & & \ddots & \\ & & & R_{t\theta_{d/2}} \end{bmatrix} $$ Here, the input row vector $$\mathbf{x}$$ is multiplied by the block diagonal matrix $$R(t)$$. This matrix is composed of $$\frac{d}{2}$$ individual 2x2 rotation matrices, $$R_{t\theta_k}$$, along its diagonal, each acting on a consecutive pair of components from the vector $$\mathbf{x}$$.

Euclidean Representation of RoPE Rotation

The Rotary Positional Embedding (RoPE) transformation can be expressed as an element-wise operation on a d-dimensional vector $$\mathbf{x}$$. This formulation calculates the rotated vector, $$\mathrm{Ro}(\mathbf{x}, t\theta)$$, by summing two vectors derived from Hadamard products (element-wise multiplication). The first vector is the product of the input vector $$\mathbf{x}$$ and a cosine vector. The second is the product of a permuted input vector ($$[-x_2, x_1, \dots, -x_d, x_{d-1}]$$) and a sine vector. The complete formula is: $$ \mathrm{Ro}(\mathbf{x}, t\theta) = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_{d-1} \\ x_d \end{pmatrix} \odot \begin{pmatrix} \cos t\theta_1 \\ \cos t\theta_1 \\ \vdots \\ \cos t\theta_{d/2} \\ \cos t\theta_{d/2} \end{pmatrix} + \begin{pmatrix} -x_2 \\ x_1 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix} \odot \begin{pmatrix} \sin t\theta_1 \\ \sin t\theta_1 \\ \vdots \\ \sin t\theta_{d/2} \\ \sin t\theta_{d/2} \end{pmatrix} $$ where $$\odot$$ denotes the Hadamard product.

Element-wise Formula for RoPE Rotation

A 2-dimensional vector can be rotated by an angle θ using two different mathematical approaches. The first approach involves multiplying the vector [x₁, x₂] by a standard 2x2 rotation matrix. The second approach treats the vector as a complex number z = x₁ + i·x₂ and multiplies it by e^(iθ).

Demonstrate mathematically that these two approaches are equivalent by showing that the components of the resulting vector from the matrix multiplication are identical to the real and imaginary parts of the resulting complex number.

Equivalence of 2D Vector Rotation Representations

Your task is to algebraically demonstrate that the two methods described below for rotating a 2D vector are equivalent. Show your work by expanding the complex multiplication in Method A and proving that its resulting real and imaginary components are identical to the formulas for the new vector components in Method B.

Verifying Equivalence of 2D Rotational Methods

A positional encoding method can be described in two ways. In one view, a d-dimensional vector is treated as d/2 complex numbers, and each complex number is rotated by an angle. In another view, the same d-dimensional vector is treated as d/2 pairs of real numbers, and each pair is rotated as a 2D vector by the same corresponding angle. What is the core principle that makes these two views equivalent?

Learn Before

Related