Google

The theoretical reward model loss, defined as an expectation, is practically implemented as an empirical loss by averaging over the collected preference dataset $$\mathcal{D}_r$$. This is based on the assumption that the data points are sampled uniformly. The formula for this empirical loss is: $$\mathcal{L}_r(\phi) = -\frac{1}{|\mathcal{D}_r|} \sum_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)\in\mathcal{D}_r} \log \mathrm{Pr}_{\phi}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})$$. Here, $$|\mathcal{D}_r|$$ represents the total number of preference pairs in the dataset.

Empirical Reward Model Loss Formula

A team is training a reward model using a dataset of 10,000 preference pairs. They notice that 2,000 of these pairs are for the single prompt, 'Write a story about a robot,' while the remaining 8,000 pairs are distributed across 4,000 other unique prompts. Given the standard empirical loss formula used for this training:

$\mathcal{L}_r(\phi) = -\frac{1}{|\mathcal{D}_r|} \sum_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)\in\mathcal{D}_r} \log \text{Pr}_{\phi}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})$

Analyze the most likely consequence of this data distribution on the trained reward model's behavior, and explain how the structure of the formula leads to this outcome.

Impact of Data Distribution on Reward Model Training

Analyze the following scenario and determine which of the two reward models is performing better on its respective dataset. Justify your answer by referencing the components of the empirical loss calculation.

Comparing Reward Model Performance

The reward model is trained by minimizing an empirical loss function derived from the Bradley-Terry model for pairwise comparisons. The objective is to adjust the model's parameters, $$\phi$$, to minimize the negative log-likelihood of the observed human preferences in the dataset $$\mathcal{D}_r$$. This is achieved by applying the sigmoid function to the difference in reward scores for the preferred response, $$\mathbf{y}_a$$, and the rejected response, $$\mathbf{y}_b$$, and then minimizing the negative logarithm of this probability, averaged over the entire dataset. The formula is: $$\min_{\phi} - \frac{1}{|\mathcal{D}_r|} \sum_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b) \in \mathcal{D}_r} \log \sigma(r_\phi(\mathbf{x},\mathbf{y}_a) - r_\phi(\mathbf{x},\mathbf{y}_b))$$

Empirical Reward Model Loss Formula using Bradley-Terry Model

A researcher is training a reward model using a small preference dataset, $\mathcal{D}_r$, which contains exactly two preference pairs:

1. For input $\mathbf{x}_1$, response $\mathbf{y}_{1a}$ is preferred over $\mathbf{y}_{1b}$.
2. For input $\mathbf{x}_2$, response $\mathbf{y}_{2a}$ is preferred over $\mathbf{y}_{2b}$.

Given the empirical loss formula $\mathcal{L}_r(\phi) = -\frac{1}{|\mathcal{D}_r|} \sum_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)\in\mathcal{D}_r} \log \text{Pr}_{\phi}(\mathbf{y

Learn Before

Related