Google

The objective in the policy learning phase of Reinforcement Learning from Human Feedback (RLHF) is to find the optimal policy parameters, denoted as $$\tilde{\theta}$$, that maximize the expected reward. The optimization process starts with the parameters of a pre-trained model, $$\hat{\theta}^{+}$$, and seeks to maximize the reward assigned by a learned reward model, $$R_{\hat{\omega}}$$. The formal expression is: $$ \tilde{\theta} = \arg \max_{\hat{\theta}^{+}} \mathbb{E}_{(\mathbf{x}, \mathbf{y}_{\hat{\theta}^{+}}) \sim \mathcal{D}_{\text{rlft}}} R_{\hat{\omega}}(\mathbf{x}, \mathbf{y}_{\hat{\theta}^{+}}) $$ Here: - $$\tilde{\theta}$$ are the optimized policy parameters. - $$\arg \max_{\hat{\theta}^{+}}$$ indicates that we are searching for the parameters that maximize the objective, starting from the initial parameters $$\hat{\theta}^{+}$$. - $$\mathbb{E}_{(\mathbf{x}, \mathbf{y}_{\hat{\theta}^{+}}) \sim \mathcal{D}_{\text{rlft}}}$$ represents the expected value over the dataset $$\mathcal{D}_{\text{rlft}}$$. For each input $$\mathbf{x}$$ from the dataset, a response $$\mathbf{y}_{\hat{\theta}^{+}}$$ is generated by the current policy. - $$R_{\hat{\omega}}(\mathbf{x}, \mathbf{y}_{\hat{\theta}^{+}})$$ is the score assigned by the reward model (with parameters $$\hat{\omega}$$) to the generated response for the given input.

Objective Function for Policy Learning in RLHF

A penalty term is incorporated into the RLHF objective function to regularize the policy and prevent it from deviating excessively from a reference policy. This penalty is formulated as the difference between the log probabilities of a sequence under the current policy ($\theta$) and the reference policy ($\theta_{ref}$), summed over all tokens in the sequence. The formula is: $Penalty = \log Pr_{\theta}(y|x) - \log Pr_{\theta_{ref}}(y|x) = \sum_{t=1}^{T} \log Pr_{\theta}(y_t|x, y_{<t}) - \sum_{t=1}^{T} \log Pr_{\theta_{ref}}(y_t|x, y_{<t})$.

KL-Divergence Penalty in RLHF Policy Optimization

Based on the provided scenario, identify which component of the optimization process is the most probable source of the model's new, undesirable behaviors. Justify your reasoning by explaining how this component directly influences the final model parameters.

Diagnosing Undesirable Model Behavior

Match each mathematical component from the policy learning objective function with its conceptual role in the training process.

A team is fine-tuning a language model where the only goal is to adjust the model's parameters to maximize the average score from a fixed reward model. After many training iterations, the team observes that while the policy consistently achieves high reward scores, the generated text is becoming repetitive and stylistically unnatural. What is the most likely reason for this outcome, based on the optimization objective?

Learn Before

Related