Google

The overall Key-Value (KV) cache generated by a Transformer's decoding network is a composite structure containing the individual KV caches from each of its layers. For a model with $$L$$ layers, the complete cache is represented as a collection of these layer-specific caches: $$\text{cache} = \{\text{cache}^1, \dots, \text{cache}^L\}$$ where $$\text{cache}^i$$ is the KV cache from the $$i$$-th layer.

Layer-wise Structure of the KV Cache

A machine learning engineer is profiling a large language model's memory usage during text generation. They observe that when they double the number of decoder layers in the model, the total memory consumed by the key-value (KV) cache also approximately doubles, even when the input text length remains the same. Based on the structural organization of the KV cache across the model's architecture, explain this observation.

KV Cache Memory Scaling

A developer is debugging a 24-layer Transformer decoder during inference. They suspect an issue with the self-attention mechanism in the 10th layer. Given that the complete KV cache is a collection of individual caches from each layer, describe the specific data structure the developer must access to inspect the keys and values computed by only the 10th layer.

Accessing a Specific Layer's KV Cache

A developer is examining the internal state of a 12-layer Transformer decoder after it has processed an input prompt. They notice that the generated Key-Value (KV) cache is not a single, large data structure, but is instead organized as a collection of 12 separate caches. What is the fundamental reason for this layer-wise organization?

Learn Before

Related