Google

The attention output for a single query vector, $$\mathbf{q}_i'$$, is computed based on the key matrix $$\mathbf{K}$$ and value matrix $$\mathbf{V}$$. This formulation calculates attention scores by taking the dot product of the query with the transposed key matrix and scaling the result by dividing by $$\sqrt{d}$$. The Softmax function converts these scores into attention weights, which are then used to produce a weighted sum of the value vectors. The formula is: $$Att_{qkv}(\mathbf{q}_i', \mathbf{K}, \mathbf{V}) = \text{Softmax}\left(\frac{\mathbf{q}_i' \mathbf{K}^T}{\sqrt{d}}\right) \mathbf{V}$$

Single-Query Scaled Dot-Product Attention

Causal attention is a type of self-attention mechanism where a query at a specific position `i` can only attend to keys and values at positions less than or equal to `i` (`K_<=i`, `V_<=i`). This restriction, often implemented using a mask, ensures that the model's prediction for a token only depends on the preceding tokens and not on future ones. The computation is formally expressed as `Att_qkv(q_i, K_<=i, V_<=i)`.

Causal Attention

An attention mechanism computes scores for a single query vector `q'` against a key matrix `K` using the formula `scores = q' * K^T * sqrt(d)`, where `d` is the vector dimension. Given `q' = [2, 1]`, `K = [[1, 0], [3, 2]]`, and `d = 4`, calculate the resulting `scores` vector.

Calculating Pre-Softmax Attention Scores

You are given a single query vector, a key matrix, a value matrix, and the vector dimension `d`. Your task is to compute the final attention output vector by applying the scaled dot-product attention formula. Show the intermediate results for each major step.

Applying Scaled Dot-Product Attention

In an attention mechanism, the scores for a query vector `q` are calculated by taking its dot product with a set of key vectors `K`. These scores are then scaled by a factor related to the vector dimension before being passed to a Softmax function to produce weights. A developer implements this but omits the scaling step, using the formula `Softmax(q * K^T) * V`. What is the most likely adverse effect of this omission, especially when the dimension of the key vectors is large?

Learn Before

Related