Google

In determining the retrieval-based distribution for a $$k$$-nearest neighbors ($$k$$-NN) language model, a distance metric, $$d_v$$, is defined relative to the vocabulary, $$V$$. For a query's hidden state, $$\mathbf{h}_i$$, and a retrieved datastore key-value pair, $$(\mathbf{z}_j, w_j)$$, the value $$d_v$$ equals the distance between $$\mathbf{h}_i$$ and $$\mathbf{z}_j$$ if the token $$w_j$$ corresponds to the $$v$$-th entry of the vocabulary $$V$$. If $$w_j$$ does not match the $$v$$-th entry, $$d_v$$ is set to $${}0$$.

Aggregated Distance Calculation for k-NN Vocabulary Distribution

In a $${}k$$-nearest neighbors ($${}k$$-NN) language model, a retrieval-based probability distribution is defined over the vocabulary $$V$$. Given a hidden state representation $$\mathbf{h}_i$$, this distribution is computed by applying the Softmax function to a vector of negative distances: $$\mathrm{Pr}_{k\mathrm{nn}}(\cdot|\mathbf{h}_i) = \mathrm{Softmax}\left(\begin{bmatrix} -d_0 & \cdots & -d_{|V|} \end{bmatrix}\right)$$. Here, $$d_v$$ represents the distance between $$\mathbf{h}_i$$ and the retrieved key $$\mathbf{z}_j$$ if the corresponding reference token $$w_j$$ matches the $$v$$-th entry of the vocabulary $$V$$; otherwise, $$d_v$$ is $${}0$$.

Softmax-based k-NN Probability Distribution

A language model is predicting the next word. It has a vocabulary of { 'cat', 'dog', 'fox', 'hen' }. After processing a context, it retrieves the 5 nearest neighbors from its datastore, listed below with their distances and associated word tokens:

*   Neighbor 1: (distance=0.2, token='dog')
*   Neighbor 2: (distance=0.4, token='fox')
*   Neighbor 3: (distance=0.5, token='cat')
*   Neighbor 4: (distance=0.7, token='dog')
*   Neighbor 5: (distance=0.9, token='fox')

Based on a method where the aggregated distance for a vocabulary token is the distance to its *closest* matching neighbor in the retrieved set, what are the aggregated distances for each token in the vocabulary? (Note: If a token does not appear in the neighbors, its distance is effectively infinite).

Calculating Aggregated Distances from Nearest Neighbors

A language model retrieves the 5 nearest neighbors from its datastore to help predict the next word. The retrieved items, consisting of a token value and its distance from the current context, are listed below. Based on this information, analyze and explain why the model's final prediction is more likely to be 'ocean' than 'sea'. Your explanation must focus on how the aggregated distance for each of these two specific words is determined from the neighbors, assuming the aggregated distance for a given word is the *minimum* distance found among the neighbors with that word as their value.

Analyzing Prediction Outcomes via Neighbor Distances

A k-NN Language Model retrieves the 4 nearest neighbors from its datastore for a given query hidden state. The retrieved neighbors, their corresponding token values, and their distances to the query are listed below:

*   Neighbor 1: Value = 'cat', Distance = 0.2
*   Neighbor 2: Value = 'dog', Distance = 0.3
*   Neighbor 3: Value = 'cat', Distance = 0.5
*   Neighbor 4: Value = 'fish', Distance = 0.6

Based on this information, what is the aggregated distance, $d_v$, for the vocabulary token 'cat

Learn Before

Related