Google

The overall objective function for training language models with Proximal Policy Optimization (PPO), denoted as $$U$$, combines the clipped surrogate objective with a policy divergence penalty. This composite objective is formulated as: $$ U(\mathbf{x}, \mathbf{y}; \theta) = U_{\text{ppo-clip}}(\mathbf{x}, \mathbf{y}; \theta) - \beta \text{Penalty} $$ In this equation, $$U_{\text{ppo-clip}}$$ represents the PPO clipped objective, while the $$\text{Penalty}$$ term quantifies the divergence from a reference policy. The hyperparameter $$\beta$$ serves as a coefficient to control the magnitude of this penalty.

Overall PPO Objective Function for Language Models

Analyze the two scenarios described in the case study below. For each scenario, predict the most likely behavior of the fine-tuned language model and explain your reasoning by referring to the components of the combined objective function used in training.

Impact of Penalty Coefficient on LLM Fine-Tuning

A researcher is training a language model using an objective function that combines a clipped surrogate objective (which encourages high rewards) with a policy divergence penalty, controlled by a coefficient β. If the researcher decides to set β to zero for the entire training process, what are the likely consequences for the model's generated text? Describe both a potential short-term benefit and a significant long-term drawback.

Consequences of Modifying the PPO Objective Function

The policy in RLHF is updated by minimizing the Proximal Policy Optimization (PPO) loss. This objective function combines a clipped surrogate objective, which uses the advantage function $$A_t$$, with a penalty term to prevent large deviations from the reference policy ($$\text{Pr}_{\theta_{\text{old}}}$$). The formula is expressed as: $$\min_{\theta} - \sum_{x \in D, y \sim \text{Pr}_{\theta_{\text{old}}}(\cdot|x)} \sum_{t=1}^{T} \left[ \text{Clip} \left( \frac{\text{Pr}_{\theta}(y_t|x, y_{<t})}{\text{Pr}_{\theta_{\text{old}}}(y_t|x, y_{<t})} \right) A_t - \beta \left( \log \text{Pr}_{\theta}(y_t|x, y_{<t}) - \log \text{Pr}_{\theta_{\text{old}}}(y_t|x, y_{<t}) \right) \right]$$ This loss is minimized over all prompts $$x$$ in the dataset $$D$$ and for each token $$t$$ in the generated sequence $$y$$. The term scaled by $$\beta$$ acts as a KL-divergence penalty to ensure training stability.

PPO Objective Formula for LLM Training in RLHF

A language model is being trained using an objective function that balances a reward-based component with a penalty for deviating from an initial reference policy. The penalty's influence is controlled by a coefficient, β. During training, developers observe that the model's outputs, while achieving high reward scores, are becoming increasingly repetitive and nonsensical. Which of the following adjustments to β is the most appropriate first step to mitigate this issue, and why?

Learn Before

Related