Google

This diagram illustrates the inference process of a k-NN Language Model. The architecture operates with two parallel streams originating from a query vector `q_i`. The first stream represents the base Large Language Model (LLM), where the query interacts with the local Key-Value (KV) cache to produce a standard probability distribution over the vocabulary, denoted as `Distribution Pr(.)`. In parallel, the second stream uses the same query to search an external datastore and retrieve its `k` nearest neighbors. These neighbors, which consist of keys and their corresponding next tokens, are then used to form a k-NN probability distribution, `Distribution Pr_knn(.)`. In the final step, these two distributions are combined to generate the final `Output Distribution` for next-token prediction.

Visual Representation of k-NN Language Model Inference

A language model architecture processes a query in two parallel streams. The first stream uses the model's internal parameters and a local context cache to generate a standard probability distribution. The second stream uses the same query to search a large external datastore, find the 'k' most similar entries, and generate a second probability distribution based on the tokens that follow those similar entries. The two distributions are then combined to produce the final output. 

Now, consider this scenario: When prompted with 'The chemical element with the atomic number 79 is...', the model provides the correct and specific completion: 'Gold'. Which of the two processing streams is most likely responsible for this high-confidence, factual recall, and why?

Analyzing Factual Recall in a Dual-Stream Language Model

A diagram of a language model's inference process shows two parallel streams originating from a single query vector. The first stream processes the query against a local cache of recent context to produce a probability distribution. The second stream uses the same query to search a large external datastore, retrieving similar past examples to form a second probability distribution. Finally, these two distributions are combined for the final prediction. What is the primary advantage of this dual-

Learn Before

Related