Google

When adapting Rotary Positional Embeddings (RoPE) for different sequence lengths, a new vector of frequency parameters, $$\theta'$$, is defined. The components of this vector are calculated using a scaling factor $$\lambda$$, a base term $$b$$, and the embedding dimension $$d$$. The formula for the vector is: $$\theta' = [(\lambda b)^{-\frac{0}{d}}, (\lambda b)^{-\frac{2}{d}}, \dots, (\lambda b)^{-\frac{d-2}{d}}]$$

Formula for Scaled RoPE Frequency Parameters (θ')

Analyze the following scenario and determine the appropriate adjustment.

Evaluating RoPE Scaling for Context Extension

Consider the formula for scaled RoPE frequency parameters: $$ \theta' = [(\lambda b)^{-\frac{0}{d}}, (\lambda b)^{-\frac{2}{d}}, \dots, (\lambda b)^{-\frac{d-2}{d}}] $$ Explain how decreasing the scaling factor `λ` (where `0 < λ < 1`) affects the individual frequency components in the vector `θ'`. What is the corresponding effect on the rotational periods of the embeddings?

Impact of Scaling Factor on RoPE Frequencies

A language model's positional embeddings are being adapted for a new context length. The adaptation uses a scaling factor `λ = 0.25`, a base term `b = 10000`, and an embedding dimension `d = 128`. Based on the formula for the scaled frequency parameters, `θ' = [ (λb)^(-0/d), (λb)^(-2/d), ..., (λb)^(-(d-2)/d) ]`, what is the approximate value of the second component in the `θ'` vector?

Learn Before

Related