Google

To give varying levels of importance to past information, a weighted moving average can be used to create summary vectors for the memory component ($$\mathrm{Mem}$$). This method applies different weights, or coefficients ($$\beta_1, \dots, \beta_{n_c}$$), to the key and value vectors within the attention window. The specific values for these coefficients can be either learned as model parameters or determined via heuristics.

Weighted Moving Average for Memory Component

The memory component, $$\mathrm{Mem}$$, can be computed using a weighted version of the moving average of the last $$n_c$$ key and value vectors. The weights, denoted by $$\beta$$, are applied to each key-value pair. This calculation is formally expressed as: $$\mathrm{Mem} = \Big( \frac{\sum_{j=i - n_c + 1}^{i} \beta_{j - i + n_c} \mathbf{k}_{j}} {\sum_{j=1}^{n_c} \beta_j} , \frac{\sum_{j=i - n_c + 1}^{i} \beta_{j - i + n_c} \mathbf{v}_{j}}{\sum_{j=1}^{n_c} \beta_j} \Big)$$

Formula for Memory as a Weighted Moving Average of Keys and Values

A common heuristic for setting the coefficients in a weighted moving average is to use values that increase for more recent items. This gives larger weight to positions that are closer to the current position $$i$$. This can be represented by a sequence of coefficients, $$\beta_j$$, that are strictly increasing, as shown by the inequality: $$\beta_1 < \beta_2 < \dots < \beta_{n_c-1} < \beta_{n_c}$$ In this setup, $$\beta_1$$ would be the coefficient for the oldest item in the moving average window, and $$\beta_{n_c}$$ would be the coefficient for the most recent item.

Increasing Coefficients as a Heuristic for Weighted Moving Average

A language model uses a weighted moving average to create a summary of past information for its memory component. The weights used in this average can either be learned as part of the model's training process or set using a pre-defined heuristic (e.g., giving more weight to recent information). Analyze the potential advantages and disadvantages of each approach (learned vs. heuristic) for determining these weights.

Learned vs. Heuristic Weights for Memory Summarization

Based on the scenario provided, how should the weights (coefficients) in the moving average be configured to best meet the requirement for long-term narrative coherence? Explain your reasoning.

Configuring Memory for Narrative Coherence

A language model's memory component creates a summary vector of past information using a weighted moving average. The weights are determined by a heuristic that assigns significantly higher importance to more recent information. For a task like summarizing a long, complex article, what is the most probable impact of this specific weighting scheme on the model's output?

Learn Before

Related