Google

The period, $$T_k$$, of the sine and cosine functions used in Rotary Positional Embeddings (RoPE) is determined by the frequency parameter $$\theta_k$$. The formula for the period is given by: $$T_k = 2\pi \cdot b^{\frac{2(k-1)}{d}}$$ where $$b$$ is the base, $$k$$ is the component index, and $$d$$ is the embedding dimensionality. This formula is derived from the relationship $$T_k = 2\pi / \theta_k$$.

Formula for the Period of RoPE's Sine and Cosine Components

A large language model uses rotary positional embeddings with an embedding dimensionality of `d = 128` and a base of `b = 10000`. Using the formula `T_k = 2π * b^(2(k-1)/d)`, calculate the period for the component with index `k = 33`.

Calculating RoPE Component Period

Given the formula for the period of the sine and cosine components, `T_k = 2π * b^(2(k-1)/d)`, what is the most plausible error in an engineer's implementation that would cause the period `T_k` to consistently *decrease* as the component index `k` increases?

Diagnosing RoPE Implementation Anomaly

A language model architect is adjusting the dimensionality `d` of a model's embeddings, which affects the periods of the sine and cosine functions used for positional information. The period `T_k` for the k-th component is calculated using the formula: `T_k = 2π * b^(2(k-1)/d)`, where `b` is a fixed base greater than 1. If the architect increases the dimensionality `d` while keeping the component index `k` (where `k > 1`) and base `b` constant, how will the period `T_k` be affected?

Learn Before

Related