Google

A common and straightforward stopping strategy in LLM inference is to terminate the generation process upon the production of a special end-of-sequence (EOS) token, such as ⟨EOS⟩ or ⟨/s⟩. Models are specifically trained to output this token to indicate that the generated text is complete.

End-of-Sequence (EOS) Token as a Stopping Criterion

A developer fine-tunes a language model on a large dataset of single, complete sentences. However, they forget to append the special token that normally signals the end of a sequence to each sentence in the training data. When this model is later used for text generation, what is the most probable and problematic behavior it will exhibit, and why does this happen?

Consequences of Training Data Omissions

Based on the common method models use to signal the natural conclusion of a response, what is the most probable technical reason for this premature termination, and what aspect of the model's training data might have led to this behavior?

Debugging Premature Text Generation Termination

A developer is using a text-generation model to complete the sentence: 'The capital of France is'. The model produces the single word 'Paris' and then immediately stops. The developer had configured the generation process to allow for a maximum of 100 new words and is surprised by the short output. Based on how these models are trained to signal completeness, what is the most likely reason the generation process terminated after just one word?

Learn Before

Related