Google

The memory state, denoted as $$\mathrm{Mem}$$, is represented as a tuple. The first element of the tuple is the arithmetic mean of key vectors ($$\mathbf{k}_j$$) and the second element is the arithmetic mean of value vectors ($$\mathbf{v}_j$$), both summed from index $$j=0$$ to $$i$$. The formula is: $$\mathrm{Mem} = \left( \frac{\sum_{j=0}^{i} \mathbf{k}_j}{i + 1}, \frac{\sum_{j=0}^{i} \mathbf{v}_j}{i + 1} \right)$$ This calculation effectively summarizes the information contained in the sequence of key-value pairs up to step $$i$$.

Formula for Memory State as an Average of Keys and Values

A memory state is calculated as a tuple containing the arithmetic mean of key vectors and the arithmetic mean of value vectors. After processing three key-value pairs (indexed 0, 1, and 2), the final memory state is `Mem = ([3, 4], [5, 6])`. Given the first two pairs are `k₀ = [1, 2]`, `v₀ = [7, 8]` and `k₁ = [2, 3]`, `v₁ = [4, 5]`, what are the vectors for the third key-value pair, `k₂` and `v₂`? Show your work.

Deriving a Missing Input from a Memory State

A system represents its memory as a tuple containing two vectors: the arithmetic mean of all key vectors seen so far, and the arithmetic mean of all value vectors seen so far. Critically evaluate this approach by discussing one significant advantage and one significant disadvantage of using a simple arithmetic mean to summarize a sequence of information.

Evaluating a Simple Memory Mechanism

Consider a sequence of three key-value pairs. The key vectors are k₀ = [1, 2], k₁ = [2, 3], and k₂ = [3, 1]. The corresponding value vectors are v₀ = [3, 0], v₁ = [1, 5], and v₂ = [2, 1]. If a memory state is defined as a tuple containing the arithmetic mean of the key vectors and the arithmetic mean of the value vectors, what is the resulting memory state after processing all three pairs?

Learn Before

Related