Google

An alternative to using a sliding window for the memory component (`Mem`) is to define it as a pair of summary vectors. This approach creates a more compressed representation of the sequence's history, rather than storing a subset of the raw key-value pairs.

Summary Vectors for Memory Compression in Attention

A straightforward method for creating summary vectors for the memory component ($$\mathrm{Mem}$$) is to compute the moving average of past keys and values. For instance, $$\mathrm{Mem}$$ can be constructed from the unweighted moving average of the last $$n_c$$ key and value vectors, providing a simple yet effective way to summarize history.

Moving Average of Keys and Values for Memory Component

To give varying levels of importance to past information, a weighted moving average can be used to create summary vectors for the memory component ($$\mathrm{Mem}$$). This method applies different weights, or coefficients ($$\beta_1, \dots, \beta_{n_c}$$), to the key and value vectors within the attention window. The specific values for these coefficients can be either learned as model parameters or determined via heuristics.

Weighted Moving Average for Memory Component

The moving average approach for creating memory summary vectors can be extended to a cumulative average of the keys and values. Instead of averaging over a fixed-size window of recent key-value pairs, this method extends the moving average to include all positions from the beginning of the sequence up to the current position $$i$$. This results in a summary that incorporates the entire history of the sequence at each step.

Cumulative Average of Keys and Values for Memory Component

A language model is designed to process extremely long documents. To manage computational costs, its attention mechanism uses a fixed-size memory component. One implementation stores the raw key-value pairs from the last 100 tokens. An alternative implementation creates a pair of 'summary vectors' by mathematically combining information from *all* preceding tokens into a fixed-size representation. Compare these two approaches in terms of the type of historical information each one preserves and the type of information each one might lose.

Memory Representation in Attention Mechanisms

A fixed-size memory cache can be maintained through a recurrent update mechanism, as depicted by a recurrent network acting as a cache. At each time step `i`, the new key-value pair, denoted as `S_kv`, is combined with the memory state from the previous step, `Mem_pre`. An `Update` function processes these two inputs to compute the new, compressed memory state, `Mem`. This approach allows the model to summarize an arbitrarily long history of key-value pairs into a constant-size memory representation, such as a single key and value pair (Size = 1 x 2).

Recurrent Update for Memory Caching

A language model is built with a memory mechanism that, to ensure constant computational cost, only stores the raw key-value pairs from the most recent 512 positions in a sequence. While processing a 10,000-word document, this model fails to recall specific details mentioned in the first few paragraphs. Based on how the memory is represented, critique the current approach and propose an alternative memory representation strategy that could mitigate this issue of losing long-range information.

Optimizing Memory for Long-Sequence Processing

An engineer is designing a language model that must process very long sequences while keeping the computational cost of attention constant at each step. They are considering two approaches for the model's memory component:

*   **Approach 1:** The memory stores the raw key-value pairs from the 256 most recent positions in the sequence.
*   **Approach 2:** The memory is a pair of fixed-size 'summary' vectors, which are calculated by mathematically combining all preceding key-value pairs into a si

Learn Before

Related