Google

In autoregressive models, output tokens, denoted by the sequence `y`, are generated in a sequential manner. This means that the prediction of any given token can only occur after all preceding tokens in the sequence have been generated first.

Sequential Generation of Output Tokens

In autoregressive models, the generation of each token is causally dependent on all previously generated tokens. This sequential dependency means that the computation for a given token cannot begin until the computation for the preceding token is complete. As a result, there is an inherent delay in predicting subsequent tokens in a sequence; for instance, the prediction of the second output token is delayed until the first has been fully generated.

Latency from Sequential Dependency in Autoregressive Generation

A developer proposes modifying a standard autoregressive language model to generate all the words of a sentence at the same time, in a single step, to make it faster. Based on the fundamental principle of how these models generate output, explain why this parallel generation approach is not possible.

Feasibility of Parallel Token Generation

An autoregressive model is tasked with generating the three-token sequence 'The cat sat'. Arrange the following computational steps in the correct chronological order.

An autoregressive model is in the process of generating the four-token sequence: `A B C D`. At the specific step where it is predicting token `D`, what information serves as the context for this prediction?

Learn Before

Related