Google

The update to the local memory component ($$\mathrm{Mem}$$) in the Compressive Transformer follows a First-In, First-Out (FIFO) rule, which is expressed with the formula: $$\mathrm{Mem} = \mathrm{FIFO}(S_{\mathrm{kv}}^{k}, \mathrm{Mem}_{\mathrm{pre}})$$. In this equation, $$\mathrm{Mem}_{\mathrm{pre}}$$ represents the memory state prior to the update. The function appends the key-value pairs from the newest segment, $$S_{\mathrm{kv}}^{k}$$, to the memory, and removes the oldest key-value pairs to maintain a fixed size, yielding the new memory state, $$\mathrm{Mem}$$.

Formula for FIFO Memory Update in Compressive Transformer

A memory component in a sequence processing model is updated using a First-In, First-Out (FIFO) rule. The memory has a fixed capacity of 3 segments. The current state of the memory, ordered from oldest to newest, is `[Segment_A, Segment_B, Segment_C]`. A new segment, `Segment_D`, is processed. According to the FIFO update rule, what will be the new state of the memory after the update?

The formula `Mem = FIFO(S_kv^k, Mem_pre)` describes an update to a fixed-size memory buffer in a sequence processing model. Arrange the operations below into the correct chronological order that this formula represents.

A sequence processing model uses the formula `Mem_new = FIFO(S_new, Mem_old)` to update its fixed-size memory. Match each component of the formula to its correct description.

Learn Before

Related