Google

The notation $$\mathbf{K}_{[i-n+1,i]}$$ is used to select a sub-matrix or a slice from a larger matrix $$\mathbf{K}$$. This slice comprises a continuous block of elements, such as rows or columns, that begins at index $$i-n+1$$ and concludes at index $$i$$. The variable $$n$$ typically represents the size of a context or window.

Matrix Slice Notation for a Sequence Window ($$\mathbf{K}_{[i-n+1,i]}$$)

A key matrix from a sliding window is a sub-matrix formed by selecting a contiguous sequence of key vectors. When denoted using slice notation as $$\mathbf{K}_{[i-n_c+1,i]}$$, it is constructed by vertically stacking the key vectors from index $$i-n_c+1$$ up to $$i$$. This structure is represented by the formula: $$\mathbf{K}_{[i-n_c+1,i]} = \begin{bmatrix} \mathbf{k}_{i-n_c+1} \\ \vdots \\ \mathbf{k}_i \end{bmatrix}$$ This matrix is a fundamental component in attention mechanisms that operate on a fixed-size context window.

Key Matrix from a Sliding Window

The notation $\mathbf{M}_{[j-9, j]}$ is used to select a slice of row vectors from a larger matrix $\mathbf{M}$. This slice contains a total of ____ row vectors.

An engineer is processing the sentence 'The quick brown fox jumps over the lazy dog'. Each word is converted into a row vector and stored in a matrix $\mathbf{E}$, where the vectors are indexed from 1 to 9 corresponding to their position in the sentence. The engineer needs to extract a context window of size 3 for the target word 'jumps'. This window must include the target word and the two words that come just before it. What is the correct slice notation to select this specific sub-matrix from $\mathbf{E}$? Explain your reasoning.

Extracting a Context Window from a Token Matrix

Consider a matrix $\mathbf{M}$ that contains 10 row vectors, indexed from 1 to 10 (i.e., $\mathbf{m}_1, \mathbf{m}_2, ..., \mathbf{m}_{10}$). The notation $\mathbf{M}_{[a, b]}$ is used to select a sub-matrix by vertically stacking the row vectors from index $a$ to index $b$, inclusive. Which of the following sub-matrices correctly represents $\mathbf{M}_{[6, 9]}$?

Learn Before

Related