Google

The ranking stage is the second step in the Top-k sampling process, following expansion. In this stage, all potential next tokens generated by the language model are sorted in descending order based on their assigned probabilities. A predefined integer, 'k', determines how many of the top-ranked tokens are kept for the next stage. All other tokens with lower probabilities are pruned, or removed from consideration. For example, in the provided diagram where k=3, the tokens 'cute' (Pr=.34), 'on' (Pr=.32), and 'sick' (Pr=.21) are kept, while 'are' (Pr=.12) and '.' (Pr=.01) are pruned.

Ranking Stage in Top-k Sampling

A developer is using a language model to generate creative story ideas. Analyze the provided case study and explain the most likely cause of the repetitive output, specifically focusing on the parameter 'k'.

Diagnosing Repetitive Text Generation

A language model generates the following potential next tokens and their probabilities: `{'for': 0.35, 'with': 0.25, 'in': 0.20, 'at': 0.15, 'on': 0.05}`. Identify the set of tokens that would be discarded (pruned) if the filtering parameter 'k' is set to 2, and the set that would be discarded if 'k' is set to 4. Explain the relationship between the value of 'k' and the size of the discarded set.

Analyzing the Impact of 'k' on Token Pruning

A language model has generated the following potential next tokens and their associated probabilities: `{'the': 0.45, 'a': 0.25, 'my': 0.15, 'your': 0.10, 'his': 0.05}`. If this list of tokens proceeds to a filtering stage where only the top 3 most probable tokens are kept for further processing, which set of tokens will be selected?

Learn Before

Related