Google

The notation $$Pr(y|x, \dots, x, y, \dots, y)$$ represents the conditional probability of a subsequent item $$y$$ occurring, given a context of preceding items, which can include both initial inputs ($$x$$'s) and previously generated outputs ($$y$$'s). This is a general way to express the core calculation in sequence generation models, where the probability of the next item depends on everything that came before it.

General Notation for Conditional Probability in Sequence Generation

In sequence-to-sequence models, the probability of generating a specific output token is conditioned on both the entire input sequence and all previously generated output tokens. This is represented by the formula $$\Pr(y_i \mid x_0, \dots, x_m, y_1, \dots, y_{i-1})$$, where $$y_i$$ is the current output token, $$(x_0, \dots, x_m)$$ is the complete input sequence, and $$(y_1, \dots, y_{i-1})$$ are the output tokens already generated. This conditional probability is the core calculation performed at each step of the auto-regressive generation process.

Conditional Probability in Sequence-to-Sequence Generation

A machine translation model is translating the English sentence 'The cat sat' into French. It has already generated the partial translation 'Le chat'. Using standard probability notation, write the expression for the probability that the *next* word to be generated is 's\'est'.

Notation for Machine Translation Probability

The expression $$\pi(y|x) \exp(r(x, y))$$ defines a method for adjusting a base probability distribution, $$\pi(y|x)$$, using a reward function, $$r(x, y)$$. The term $$\pi(y|x)$$ represents the initial probability of an output $$y$$ given an input $$x$$. This probability is then multiplied by $$\exp(r(x, y))$$, the exponential of the reward associated with that specific input-output pair. This re-weighting mechanism increases the likelihood of outputs with higher rewards.

Formula for Re-weighting a Probability Distribution with a Reward Function

Based on the general notation for conditional probability in sequence generation, write the precise mathematical expression for the probability that the model's next generated word is 'planet', given the full input text and the summary generated so far.

Applying Conditional Probability Notation in Text Summarization

A language model is generating text and has so far produced the sequence 'The sky is'. The model now needs to calculate the likelihood of the next word being 'blue'. Which of the following mathematical expressions correctly represents the probability of the next word being 'blue', given the preceding words?

Learn Before

Related