Google

In an attention mechanism, the Key vector (K) is associated with each item in the input sequence. It serves as a label or identifier that is compared with the Query vector. The similarity between a Query and a Key determines the attention score, or weight, assigned to the corresponding input item.

Key Vector

Based on the scenario, analyze the most likely deficiency in the 'label' vectors associated with each word in the sentence that would cause this specific failure.

Diagnosing Attention Mechanism Failure

In a system designed to focus on relevant parts of an input sentence, each word is assigned a 'label' vector that acts as its identifier. A 'context' vector, representing the current point of focus, is then compared against every word's 'label' vector to calculate a relevance score. If, due to a system error, every word in the sentence 'The quick brown fox jumps' was assigned an identical 'label' vector, what would be the consequence for the calculated relevance scores? Explain your reasoning.

Consequences of Non-Unique Identifiers in Attention

The notation $$\left\{ \mathbf{k}_{i-n_c+1}, \dots, \mathbf{k}_i \right\}$$ represents a set of key vectors, denoted by the bold letter $$\mathbf{k}$$. This set comprises all key vectors within a specific context window, starting from the index $$i-n_c+1$$ and ending at the index $$i$$. The size of this context window is determined by $$n_c$$.

Context Window of Key Vectors Notation

A language model is processing the sentence: 'The delivery driver carefully parked the large van.' To understand the context of the word 'parked', the model generates a 'focus' vector for it. This 'focus' vector is then compared against a 'label' vector for every word in the sentence to calculate relevance scores. What is the primary function of the 'label' vector associated with the word 'van' in this process?

Learn Before

Related