Google

In the context of training a Large Language Model (LLM), the optimal parameters, denoted as $$\hat{\theta}$$, are found by maximizing the conditional log-likelihood across a dataset $$D$$. This supervised learning objective involves finding the parameters $$\theta$$ that maximize the sum of the logarithmic probabilities of the true outputs $$y$$ given the inputs $$x$$, where the probability $$\text{Pr}_{\theta}(y|x)$$ is predicted by the LLM. The formula is expressed as: $$\hat{\theta} = \underset{\theta}{\arg\max} \sum_{(x,y) \in D} \log \text{Pr}_{\theta}(y|x)$$ In some contexts, the input $$x$$ can be represented by other variables, such as a context $$c$$ and a latent variable $$z$$, leading to an equivalent formulation: $$\hat{\theta} = \underset{\theta}{\arg\max} \sum_{(x,y) \in D} \log \text{Pr}_{\theta}(y|c, z)$$

Parameter Estimation via Conditional Log-Likelihood Maximization

When applying reinforcement learning to sequence generation tasks, the language model itself is treated as the policy. The policy, denoted as `π_θ`, defines the probability of choosing the next token `y_t` given the input `X` and the previously generated tokens `y_<t`. This policy is directly equivalent to the conditional probability distribution of the language model, `Pr_θ`. The relationship is formally stated as: $$ \pi_{\theta}(y_t|\mathbf{X}, \mathbf{y}_{<t}) = \text{Pr}_{\theta}(y_t|\mathbf{X}, \mathbf{y}_{<t}) $$

Language Model as a Stochastic Policy

The Plackett-Luce loss function is derived directly from the log-probability of a ground-truth ranked sequence. It is defined as the negative log-likelihood of observing this correct sequence, a formulation rooted in the principle of maximum likelihood estimation. The training objective is to minimize this loss, which effectively maximizes the model's probability of predicting the correct rankings. This loss is typically averaged over all samples in a dataset by taking the expectation of the negative log-probability.

Plackett-Luce Loss Function

A language model is being trained on a small dataset for a classification task. You are evaluating two different sets of model parameters, $\theta_A$ and $\theta_B$. For a batch of two examples, the models produce the following probabilities for the correct target classes:

**Example 1:**
- Model A: $P(y^{(1)}|x^{(1)};\theta_A) = 0.7$
- Model B: $P(y^{(1)}|x^{(1)};\theta_B) = 0.9$

**Example 2:**
- Model A: $P(y^{(2)}|x^{(2)};\theta_A) = 0.6$
- Model B: $P(y^{(2)}|x^{(2)};\theta_B) = 0.4$

Calculate the total conditional log-likelihood for the batch for each set of parameters. Based on your calculation, which set of parameters ($\theta_A$ or $\theta_B$) is considered a better fit for this data? Use the natural logarithm (ln) for your calculations.

Model Comparison using Conditional Log-Likelihood

A data scientist is training a model. After a single training update, the model's predicted probabilities for the correct outputs on two specific examples from the training dataset have changed as shown below. Based on the training objective of maximizing the sum of conditional log-likelihoods, did this specific update improve the model with respect to these two examples? Justify your answer with a calculation.

Evaluating a Training Update

A model is being trained by maximizing the sum of log-probabilities for a dataset of 1,000 examples. Consider two scenarios for a single training update:

Scenario A: The probability assigned to the correct output for one example improves from 0.1 to 0.2. The probabilities for all other 999 examples remain unchanged.

Scenario B: The probability assigned to the correct output for one example improves from 0.8 to 0.9. The probabilities for all other 999 examples remain unchanged.

Which scenario 

Learn Before

Related