Google

This diagram illustrates the fourth iteration in a continuous batching process, where a new request, x₃, arrives. The scheduler incorporates this new request into the batch. In this single computational step, the system performs two distinct operations concurrently: the prefilling phase for the new request x₃ and a single decoding step for the ongoing requests x₁ and x₂. This interleaving of compute-intensive prefilling with memory-bound decoding is a core feature of continuous batching that enhances system throughput.

Example of Concurrent Prefilling and Decoding in Continuous Batching (Iteration 4)

This diagram illustrates the fifth iteration in a continuous batching example. In this step, the scheduler directs the inference engine to perform a single decoding operation for all three active requests: x₁, x₂, and x₃. As a result of this step, the generation for request x₂ is finished, which is indicated by the 'complete' status. The full output for this request, y₂, is now available. This event signals to the scheduler that x₂ can be removed from the batch in the next iteration, freeing up resources and demonstrating a key advantage of continuous batching—the dynamic management of the batch composition.

Example of a Request Completing in Continuous Batching (Iteration 5)

Based on the following scenario, evaluate the junior engineer's argument. Is their conclusion about inefficiency correct for a system designed for high throughput? Justify your reasoning by describing the two different types of operations that can be performed concurrently in this single step.

LLM Inference Scheduling Decision

An LLM inference system is actively generating tokens for several ongoing requests. A new request with a long prompt arrives. The system's scheduler decides to process the initial, computationally-heavy phase for the new request at the same time it generates the next single token for the ongoing requests. Explain why this concurrent approach is more efficient for the overall system than processing these two types of tasks sequentially.

Efficiency of Concurrent LLM Operations

An LLM inference system is actively generating tokens for two separate user requests that are already in progress. A third user submits a new request to the system. To maximize overall throughput by overlapping different types of computation, what actions will the system perform in the next single computational step?

Learn Before

Related