Google

When processing a new input sequence $$\mathbf{x}'$$, the system checks if it shares a common prefix with a previously cached sequence. If the new input has a matching prefix $$\mathbf{x}'_{<k} = \mathbf{x}_{<k}$$ for some length $$k$$, the corresponding Key-Value (KV) cache state, $$\mathrm{cache}_{<k}$$, is loaded directly. This state is used to initialize the KV cache, allowing the model to bypass redundant computation and only compute the hidden states for the remaining subsequent tokens $$\mathbf{x}'_{\ge k}$$.

Process of Utilizing a Prefix Cache

An LLM inference system has previously processed and cached the internal states for the 8-token sequence: `Analyze the economic impact of renewable energy sources`. A new request arrives with the 7-token sequence: `Analyze the economic impact of solar power`. The common prefix is `Analyze the economic impact of` (6 tokens). Describe the key difference in the computational steps the model takes to process the new request if it successfully utilizes the prefix cache, compared to processing it without the cache. Specifically, which tokens require new computation in the cached scenario?

Computational Efficiency of Prefix Cache Utilization

A new input sequence is provided to a language model that uses a prefix cache for inference. Arrange the following steps in the correct chronological order to describe how the system utilizes the cache to process this new sequence.

An inference system for a large model has previously processed the input 'The best movie of all time is' and has stored the corresponding internal states in a cache. A new user then submits the input 'The best movie of the year is'. How will the system most efficiently use the cache to process this new request?

Learn Before

Related