Google

The memory component, $$\mathrm{Mem}$$, can be computed using a weighted version of the moving average of the last $$n_c$$ key and value vectors. The weights, denoted by $$\beta$$, are applied to each key-value pair. This calculation is formally expressed as: $$\mathrm{Mem} = \Big( \frac{\sum_{j=i - n_c + 1}^{i} \beta_{j - i + n_c} \mathbf{k}_{j}} {\sum_{j=1}^{n_c} \beta_j} , \frac{\sum_{j=i - n_c + 1}^{i} \beta_{j - i + n_c} \mathbf{v}_{j}}{\sum_{j=1}^{n_c} \beta_j} \Big)$$

Formula for Memory as a Weighted Moving Average of Keys and Values

A team is developing a language model that uses a memory component to keep track of recent information. The memory is calculated as a pair of summary vectors using the following weighted moving average formula over the last `n_c` key (`k`) and value (`v`) vectors at position `i`:

`Mem = ( (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j), (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} v_j) / (Σ_{j=1}^{n_c} β_j) )`

The team's goal is for the model to assign greater importance to the most recent information within its memory window compared to older information. Based on the structure of this formula, describe the characteristic that the weight vector `β = [β_1, β_2, ..., β_{n_c}]` should have to achieve this goal, and explain your reasoning.

Configuring Memory Component Weights

A memory component is calculated as a weighted moving average of key vectors (`k`) and value vectors (`v`). The formula for the summary key vector is given by:

`Summary Key = (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j)`

Where `i` is the current position, `n_c` is the context window size, and `β` is a vector of weights.

Given the following:
- Current position `i = 5`
- Context window size `n_c = 3`
- Weights `β = [β_1, β_2, β_3] = [0.1, 0.3, 0.6]`
- Key vectors:
  - `k_3 = [2, 4]`
  - `k_4 = [1, 3]`
  - `k_5 = [5, 0]`

Calculate the summary key vector. Provide your answer as a vector (e.g., `[x, y]`).

Calculating the Memory Summary Vector

A model computes a memory component, `Mem`, using the following formula for a weighted moving average of the last `n_c` key (`k`) and value (`v`) vectors at a given position `i`:

`Mem = ( (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j), (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} v_j) / (Σ_{j=1}^{n_c} β_j) )`

Given a current position `i=10`, a context window size `n_c=4`, and weights `β = [β_1, β_2, β_3, β_4]`, which of the following expressions correctly represents the calculation for the summar

Learn Before

Related