Google

The Reinforcement Learning from Human Feedback (RLHF) process, when implemented with Proximal Policy Optimization (PPO), involves a sequence of stages. The process starts with collecting preference data (e.g., ya ≻ yb), which is used to train a reward model. This reward model subsequently informs a value function. The final stage is policy training, where PPO is used to optimize the policy, which itself may have been initialized through Maximum Likelihood Estimation (MLE).

High-Level Process of RLHF with PPO

A language model is being fine-tuned using a reinforcement learning approach that incorporates human feedback. Arrange the following key stages of this process into the correct chronological order.

A development team is fine-tuning a large language model to be a better conversational assistant. They have already collected a dataset of human preferences, where evaluators chose the better of two model-generated responses for thousands of different prompts. Using this data, they have successfully trained a 'reward model' that accurately predicts a scalar score representing how much a human would likely prefer a given response. The team is now ready for the final stage of the process: using this reward model to update the conversational assistant itself. What is the primary goal of this final stage, and how is the scalar score from the reward model utilized to achieve this goal?

Applying a Preference Model for AI Fine-Tuning

A team is fine-tuning a language model using a reinforcement learning process guided by human feedback. They observe that while the model's policy is successfully optimized to achieve high scores from its internal reward signal, the generated text is often repetitive, nonsensical, and misaligned with the original human preferences. Which of the following is the most likely cause of this discrepancy?

Learn Before

Related