Google

The ALiBi method modifies the standard attention score by adding a positional bias term, $$\mathrm{PE}(i, j)$$, directly to the scaled query-key dot product. This integration of the linear bias into the attention calculation results in the following formula for the pre-Softmax score: $$s_{i,j} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d}} + \mathrm{PE}(i, j)$$

Formula for Attention Score with ALiBi Bias

The final attention weight in the ALiBi framework, denoted as $$\alpha(i, j)$$, is computed by applying the Softmax function to the attention score. This score is derived by adding the ALiBi positional bias term, $$\beta \cdot (j - i)$$, to the standard query-key product $$\mathbf{q}_i \mathbf{k}_j^{\mathrm{T}}$$, scaling the sum by the inverse square root of the dimension $$d$$, and incorporating an optional mask. The complete equation is expressed as: $$\alpha(i,j) = \mathrm{Softmax}\left(\frac{\mathbf{q}_i \mathbf{k}_{j}^{\mathrm{T}} + \beta \cdot (j - i)}{\sqrt{d}} + \mathrm{Mask}(i,j)\right)$$ In this formula, $$\mathbf{q}_i$$ and $$\mathbf{k}_j$$ denote the query and key vectors, and $$\beta$$ acts as a scaling factor. The $$\mathrm{Mask}(i, j)$$ term ensures proper attention masking when required.

Complete ALiBi Attention Formula

In a transformer model that incorporates a linear positional bias directly into its attention mechanism, you need to compute the pre-Softmax attention score between a query vector at position 3 and a key vector at position 1. The formula for this score involves adding a positional bias term to the standard scaled dot-product of the query and key. Use the provided information to calculate this final score.

Calculating a Pre-Softmax Attention Score with Linear Bias

In a system that calculates a pre-Softmax attention score by adding a linear positional bias to the scaled query-key dot product, describe a scenario where a key that is semantically less similar to a query (i.e., has a lower dot-product score) could receive a higher final attention score than a key that is semantically more similar. Explain your reasoning by referencing the components of the calculation.

Interaction of Semantic and Positional Scores

In a model that adds a linear positional bias to its attention calculation, a query at position `i=10` attends to two keys at positions `j1=5` and `j2=2`. Assuming the scaled dot-product portion of the score is identical for both keys, how will the addition of the positional bias term `PE(i, j)` affect their final pre-Softmax attention scores?

Learn Before

Related