Google

The loss function for the policy learning stage in RLHF is defined as the negative expected utility of the model's outputs. The objective is to find the policy parameters $\theta$ that minimize this loss, which is equivalent to maximizing the expected utility. The formula is: 

$\mathcal{L}(\theta) = -\mathbb{E}_{\mathbf{x}\sim\mathcal{D}} \mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(\cdot|\mathbf{x})} [U(\mathbf{x}, \mathbf{y}; \theta)]$ 

Where:
- $\mathcal{D}$ denotes the input-only dataset.
- $\mathbf{y} \sim \pi_{\theta}(\cdot|\mathbf{x})$ signifies that the output $\mathbf{y}$ is sampled from the probability distribution defined by the language model's policy, $\pi_{\theta}$, given the input $\mathbf{x}$.
- $U(\mathbf{x}, \mathbf{y}; \theta)$ is a utility function that scores the quality of the output $\mathbf{y}$ for the input $\mathbf{x}$.

Policy Learning Loss Function in RLHF

The primary goal when refining a language model's policy is to maximize the expected utility of its generated outputs. However, the loss function for this optimization process is typically defined as the *negative* of the expected utility. Explain the fundamental reason for this specific formulation. Why is minimizing the negative expected utility equivalent to maximizing the expected utility, and why is this approach commonly used in training machine learning models?

Rationale for the Negative Expected Utility Loss Function

Analyze the following scenario and predict the outcome of the model training. Explain your reasoning.

Consequences of an Incorrect Loss Function Implementation

An engineer is training a language model where the training objective is to adjust the model's parameters to maximize a utility score for its generated outputs. The loss function is defined as the negative of the expected utility score. During a training run, the engineer observes that the calculated loss value is consistently increasing over several iterations (e.g., moving from -15.0 to -12.5 to -10.0). What is the most direct interpretation of this observation?

Learn Before

Related