Google

In auto-regressive language modeling, text generation is an iterative process. At every step, after the model generates an output token, this new token is appended to the existing input sequence. This updated, longer sequence then serves as the context for predicting the subsequent token, effectively shifting the model's focus one position forward.

Context Shifting in Auto-Regressive Generation

An auto-regressive language model generates text one token at a time. Arrange the following actions into the correct chronological sequence that describes the process of generating one token and preparing for the next.

A developer is implementing a text generation loop for an auto-regressive language model. They notice that when they provide the input 'The ocean is vast and', the model produces the output 'deep deep deep deep...'. Based on your understanding of the iterative generation process, analyze this behavior and identify the most probable error in the developer's implementation of the generation loop.

Debugging a Repetitive Generation Loop

An auto-regressive language model is provided with the input sequence 'The sun is shining and the'. The model's next step is to generate the token ' birds'. To predict the token that will come after ' birds', what complete sequence will the model use as its new input?

Learn Before

Related