Google

In the $$k$$-nearest neighbors ($$k$$-NN) retrieval process, a datastore of key-value pairs, $$\mathbf{k}_j$$ and $$\mathbf{v}_j$$, is maintained, often within a vector database. For a given query $$\mathbf{q}_i$$, the system identifies its $$k$$ nearest neighbors from the set of keys by conceptually expanding a sphere centered at $$\mathbf{q}_i$$ until it encompasses exactly $$k$$ data points in $$\mathbf{k}_j$$. This search results in a retrieved long-term memory set, denoted as $$\mathrm{Mem}_{k\mathrm{nn}}$$, which contains the $$k$$ keys along with their corresponding values.

k-NN Memory Retrieval

k-NN search augmented attention is an architecture that processes a query vector, `qi`, through two parallel computational streams. The first stream computes standard attention over a local memory, such as the KV cache, to capture immediate context. Concurrently, the second stream uses `qi` to perform a k-NN search on an external datastore, retrieving the k nearest neighbors. A separate attention calculation is then performed over these retrieved neighbors. Finally, the outputs from these two streams are integrated to produce a final result. This integration can occur at different levels, for instance, by combining the attention output vectors directly or by converting each stream's output into a probability distribution and then merging them.

k-NN Search Augmented Attention

Analyze the following scenario and identify the single parameter of the retrieval process that is most likely causing the described issue. Propose a specific adjustment to this parameter and explain your reasoning.

Optimizing a Chatbot's Retrieval System

A system is designed to retrieve information from a datastore of key-value pairs using a nearest-neighbor approach. Arrange the following steps of this retrieval process in the correct logical sequence.

In a memory retrieval system, a query is compared against a large datastore of key-value pairs to find the 'k' most similar keys. The corresponding key-value pairs are then returned. What is the primary effect of increasing the value of 'k'?

Learn Before

Related