Google

This diagram illustrates a step in speculative decoding following the acceptance of draft tokens. Given a context `(x, yi)`, a draft model `Pr_q(·)` has generated three candidate tokens: `ˆy_{i+1}, ˆy_{i+2}, ˆy_{i+3}`. After these three tokens are accepted, the evaluation model `Pr_p(·)` is then used to predict the subsequent token, `¯y_{i+4}`. This demonstrates the process of extending the sequence after a successful speculation.

Diagram of Post-Acceptance Token Prediction in Speculative Decoding

In a text generation process, a fast draft model proposes the token sequence `['over', 'the', 'lazy']`. A more accurate evaluation model accepts the first two tokens, `'over'` and `'the'`, but rejects the third token, `'lazy'`. To continue generating the text, which model should be used to determine the very next token, and based on what preceding text sequence?

Correcting a Step in a Generation Process

In a speculative decoding process, a draft model proposes a 3-token sequence, and the main evaluation model accepts all three tokens. Arrange the following actions in the correct chronological order to describe how the very next token is generated.

In a single step of a text generation process, a small, fast model proposes the candidate token sequence `['on', 'the', 'mat']` to extend the existing text `['The', 'cat', 'sat']`. A larger, more accurate model then evaluates these candidates. The larger model accepts `'on'` and `'the'`, but rejects `'mat'`. After this rejection, the larger model's own prediction for the next token is `'rug'`. What is the complete sequence of new tokens added to the text in this step?

Learn Before

Related