Google

The problem of determining how many tokens to generate from a single masked span is conceptually similar to fertility modeling in the field of machine translation.

Analogy between Span Length Prediction and Fertility Modeling

Scenario A: A text-infilling model is tasked with completing the sentence 'The dog ___ in the park.' It replaces the blank, which could represent any number of words (e.g., 'played', 'was chasing a ball'), with a single special symbol. The model must then generate an appropriate sequence of words to replace that one symbol.

Scenario B: An early machine translation system is translating a French sentence containing the word 'au' into English. Depending on the context, this single French word might need to be translated into two English words, 'to the'.

Explain the fundamental modeling challenge that is common to both of these scenarios.

Conceptual Parallel in Generative Modeling

In some text generation tasks, a model must replace a single placeholder token with a sequence of multiple words of unknown length. This challenge is often compared to 'fertility modeling' from traditional machine translation, where a model must decide how many words in a target language correspond to a single word in the source language. Critically evaluate this analogy. In your response, discuss at least one major strength and one significant limitation of this comparison.

Critique of the Fertility Modeling Analogy

In generative language models, the challenge of predicting the length of a masked span is often compared to the concept of 'fertility' from statistical machine translation. Which of the following statements most accurately analyzes the relationship between these two concepts?

Learn Before

Related