Google

This diagram illustrates a common scenario in LLM serving where a batch contains sequences of varying lengths, such as `⟨SOS⟩ I think this movie is better` and `I really like`. To create a uniform batch for processing, the shorter sequence is prepended with a start-of-sequence token and padded with `⟨pad⟩` tokens, resulting in `⟨pad⟩ ⟨pad⟩ ⟨pad⟩ ⟨SOS⟩ I really like`. Crucially, the image also depicts how these sequences' data blocks are stored in non-contiguous physical memory, visualizing the problem of memory fragmentation that arises from dynamic allocation.

Example of Padded Sequences in Fragmented Memory

Based on the provided scenario, explain the most likely reason for the memory allocation failures and clarify why the process of equalizing sequence lengths does not prevent this issue.

Inference Server Memory Allocation Failure

A machine learning system processes text by grouping multiple sentences of varying lengths into a single batch. To ensure a uniform data structure for parallel processing, shorter sentences are extended with special `⟨pad⟩` tokens. Despite this uniform structure, the system's performance degrades over time due to memory allocation issues. Based on this scenario, explain the underlying cause of the memory inefficiency and how the padding process contributes to it.

Relationship Between Sequence Padding and Memory Inefficiency

Consider a system processing two text sequences of different lengths in a single batch. To create a uniform input, the shorter sequence is extended with special `⟨pad⟩` tokens. A visualization of the system's memory reveals that the data blocks for these sequences are stored in non-contiguous physical locations, with gaps of unused memory between them. What is the primary operational challenge illustrated by this non-contiguous storage arrangement?

Learn Before

Related