Google

In the RLHF framework, the Target Model, also known as the policy model, is the Large Language Model being actively trained. Its policy, denoted as $$\pi_{\theta}(\cdot)$$ and formally defined as the probability distribution $$\text{Pr}_{\theta}(\cdot)$$, governs the generation of the next token based on the current context. The model's parameters, $$\theta$$, are updated during training under the guidance of both the reward and value models.

Target Model (Policy Model) in RLHF

An AI development team is using a reinforcement learning process to improve a language model's ability to generate helpful and harmless responses. They observe that the model's generated text is becoming more aligned with desired outcomes over time. Based on the principles of this training process, explain the specific role of the model being actively trained (the 'policy model') and describe how its internal parameters are adjusted to achieve this improvement.

Mechanism of Policy Model Refinement

Given the following scenario, analyze the model's behavior. What specific function of this model is likely causing the observed predictability, and why is this observation critical for the training process?

Analyzing Policy Model Behavior

An engineering team is refining a large language model. During one step of the process, the model is given the start of a sentence, 'The best way to learn a new skill is'. The model then calculates a probability for every word in its vocabulary to be the next word and uses this distribution to generate a complete sentence. The model's internal parameters are then updated based on a separate quality assessment of the generated sentence. Which part of this process best describes the primary role o

Learn Before

Related