Google

The formula for single-head self-attention calculates the output for a single query vector $$\mathbf{q}_{i'}$$ based on a set of key-value pairs. The formula is: $$ \text{Att}_{\text{qkv}}(\mathbf{q}_{i'}, \mathbf{K}, \mathbf{V}) = \text{Softmax}(\frac{\mathbf{q}_{i'}\mathbf{K}^{\text{T}}}{\sqrt{d}}) \mathbf{V} $$ In this equation, the value matrix $$\mathbf{V}$$ is an element of the set of real-numbered matrices with dimensions $$i' \times d$$, expressed as $$\mathbf{V} \in \mathbb{R}^{i' \times d}$$. Here, $$i'$$ represents the number of key-value pairs in the sequence, and $$d$$ is the dimension of each value vector. The overall process involves computing the dot product of the query with all keys, scaling by the square root of the key dimension, applying the Softmax function to obtain attention weights, and finally, computing a weighted sum of the value vectors.

Formula for Single-Head Self-Attention

A computational mechanism is used to determine the relevance of different parts of an input sequence to a specific element. This involves several steps to produce a final output vector. Arrange the following computational steps in the correct chronological order.

Given the scenario below, which of the 'value' vectors will the final output vector most closely resemble? Justify your answer by describing how the intermediate steps of the calculation (specifically, the function that normalizes scores into weights and the final weighted sum) lead to this result.

Attention Mechanism Output Analysis

In a mechanism that calculates attention, scores are computed by taking the dot product of a query vector with a set of key vectors. These scores are then scaled by dividing by the square root of the dimension of the vectors (e.g., $\sqrt{d}$) before being passed to a Softmax function. What is the most likely adverse consequence of removing this scaling step, particularly when the vector dimension is large?

Learn Before

Related