Google

The notation for a key-value pair, such as $$(k', v')$$, represents a fundamental unit of information in attention mechanisms. It consists of a key vector ($$k'$$) for calculating relevance and a corresponding value vector ($$v'$$) that holds the content to be retrieved. A collection of these pairs, often denoted as $$\{(K, V), \dots, (K, V)\}$$, forms a memory component that a model can query.

Notation for Key-Value Pairs

Imagine you are designing a simple recipe retrieval system. The system's memory is structured as a collection of pairs, `{(K, V)}`, where each pair represents one recipe. When a user enters a query like 'spicy chicken pasta', the system compares the query to each `K` to find relevant recipes and then returns the corresponding `V` for the best matches. In this scenario, what would be the most logical choice for the content of `K` and the content of `V`?

Applying Key-Value Structure to a Retrieval System

Match each notational component with its correct description in the context of a memory system used for information retrieval.

The notation $$\{(K_1,V_1),\dots,(K_n,V_n)\}$$ represents a collection of key-value pairs. In each pair $$(K_j,V_j)$$, $$K_j$$ is the key and $$V_j$$ is its corresponding value; the index $$j$$ distinguishes the pairs. In attention contexts, the indexed pairs may correspond to different attention heads or layers.

Set of Key-Value Pairs

A computational system uses a collection of information structured as a set of pairs: `{(Key_1, Value_1), (Key_2, Value_2), ..., (Key_n, Value_n)}`. To respond to a new query, the system first compares the query to each `Key` to compute a relevance score for every pair. Based on this process, what is the distinct role of the `Value` components?

Learn Before

Related