Google

The matrices $$\mathbf{W}_j^{q}$$, $$\mathbf{W}_j^{k}$$, and $$\mathbf{W}_j^{v}$$ are the parameter matrices that define the transformations used within the self-attention mechanism of a Transformer model. These matrices, which belong to $$\mathbb{R}^{d \times \frac{d}{\tau}}$$, transform the input representation $$\mathbf{H}$$ into queries, keys, and values through the equations: $$\mathbf{Q}^{[j]} = \mathbf{H} \mathbf{W}_j^{q}$$, $$\mathbf{K}^{[j]} = \mathbf{H} \mathbf{W}_j^{k}$$, and $$\mathbf{V}^{[j]} = \mathbf{H} \mathbf{W}_j^{v}$$.

Parameter Matrices for Attention Transformations

After introducing weight matrices to transformers, the output calculation for yi is now based on a weighted sum over the value vectors. To avoid effective loss of gradients during training, the dot product needs to be scaled in a suitable fashion.
$score(x_{i},x_{j}) = \frac{q_{i}·k_{j}}{√d_{k}}$, where $q_{i}$ is the query vector, $k_{i}$ is the preceding element’s key vectors, and ${d_k}$ is the dimensionality of the query and key vectors. Taking this one step further, we can scale these scores, take the softmax, and then multiply the result by V resulting in a matrix of shape N ×d: a vector embedding representation for each token in the input. And we'll get the self-attention of transformer from the previous node's above node.
Since at each layer we need to compute dot products between each pair of tokens in the input, it is extremely expensive for the input to a transformer to consist of long documents.

Introduce weight matrices in the transformer

In a self-attention layer, the Query (Q), Key (K), and Value (V) vectors are not the direct inputs themselves but are generated through linear transformations of the same input sequence. This input is typically the output from the preceding layer. Each vector in the input sequence is multiplied by three distinct weight matrices ($W_q$, $W_k$, and $W_v$) to produce its corresponding Q, K, and V vectors.

Generation of Query, Key, and Value Vectors in Self-Attention

The formula $$W_j^v \in \mathbb{R}^{d \times \frac{d}{\tau}}$$ defines the value weight matrix, $$W_j^v$$. This matrix consists of real numbers and has dimensions of $$d$$ rows and $$\frac{d}{\tau}$$ columns. In the context of attention mechanisms, this matrix is used to transform the input values.

Value Weight Matrix $$W_j^v$$

This formula defines the value weight matrix $$\mathbf{W}_j^v \in \mathbb{R}^{d \times \frac{d}{\tau}}$$. In multi-head attention mechanisms, the superscript $$v$$ indicates a value projection matrix, and the subscript $$j$$ denotes the $$j$$-th attention head. The matrix transforms an input representation of dimension $$d$$ (the model's embedding dimension) into a value representation of reduced dimension $$\frac{d}{\tau}$$, where $$\tau$$ represents the number of attention heads (or a scaling factor).

Value Weight Matrix Definition ($$\mathbf{W}_j^v \in \mathbb{R}^{d \times \frac{d}{\tau}}$$)

In a self-attention mechanism, instead of directly comparing the raw input vectors of a sequence, each input vector is first multiplied by three separate, learned parameter matrices. This process creates three distinct representations of the original vector before they are used to calculate attention scores and output values. What is the primary analytical advantage of this approach over simply comparing the original input vectors to each other?

Learn Before

Related