Google

The cumulative average for the memory component can be calculated efficiently using a recursive formula. The memory state at the current step, $$\mathrm{Mem}_i$$, is computed based on the current key-value pair $$(\mathbf{k}_i, \mathbf{v}_i)$$ and the memory state from the previous step, $$\mathrm{Mem}_{i-1}$$. The recursive formula is expressed as: $$\mathrm{Mem}_i = \frac{(\mathbf{k}_i, \mathbf{v}_i) + i \cdot \mathrm{Mem}_{i-1}}{i + 1}$$ This approach prevents the need to recalculate the sum of all historical key-value pairs at each step, significantly increasing computational efficiency.

Recursive Formula for Memory as a Cumulative Average

A significant advantage of computing the cumulative average for the memory component with a recursive formula is its memory efficiency. During inference, this method only requires storing the single key-value pair representing the previous state's average, rather than retaining the entire history of all key-value pairs.

Memory Efficiency of Recursive Cumulative Average

A system updates its memory state at each step by calculating the cumulative average of all key-value pairs seen so far. It uses the following recursive formula, where 'i' is the 0-indexed step number:

Mem_i = ( (k_i, v_i) + i * Mem_{i-1} ) / (i + 1)

Suppose that after processing 5 pairs (at step i=4), the memory state is Mem_4 = (10, 20). At the next step (i=5), the system observes a new key-value pair (k_5, v_5) = (16, 22).

What is the new memory state, Mem_5? Provide your answer as a pair of numbers (x, y), rounding to two decimal places if necessary.

Calculating a Recursive Memory State

A system updates its memory state by calculating the cumulative average of all key-value pairs it has processed. It uses the following recursive formula, where `i` is the index of the current update step (e.g., for the second item, i=1): 

`Mem_i = ((k_i, v_i) + i * Mem_{i-1}) / (i+1)`

Given the sequence of inputs below, calculate the final memory state `Mem_2`.

Step-by-Step Memory State Calculation

A key advantage of using a recursive formula for the cumulative average memory model is its efficiency during inference. Because the new memory state can be calculated using only the previous state and the current key-value pair, the model does not need to store the entire history of all preceding key-value pairs. This reduces the memory requirement to a single key-value pair, making it highly efficient for long sequences.

Inference Efficiency of Cumulative Average Memory

A system processes a very long sequence of data pairs, one at a time. At each step, it must update a 'memory state' to be the precise mathematical average of all data pairs seen up to that point. Consider two methods for updating the memory state at the 10,000th step:

Method A: Re-access the entire history of 10,000 data pairs, sum them up, and divide by 10,000.

Method B: Use only the memory state from the 9,999th step (which was the average of the first 9,999 pairs) and the new 10,000th data 

Learn Before

Related