Google

The value function, parameterized by $$\omega$$, is trained alongside the policy to estimate the expected future reward from a given state. Its parameters are updated by minimizing the Mean Squared Error (MSE) between the predicted state value, $$V_\omega(\mathbf{x},y_{<t})$$, and the computed return. The computed return is the sum of the immediate reward, $$r_t$$, and the discounted value of the next state, $$\gamma V_\omega(\mathbf{x},y_{<t+1})$$. The loss function is averaged over a dataset $$\mathcal{D}$$ and all token positions $$T$$: $$\min_{\omega} \frac{1}{M} \sum_{\mathbf{x} \in \mathcal{D}} \sum_{t=1}^{T} \left( r_t + \gamma V_\omega(\mathbf{x},y_{<t+1}) - V_\omega(\mathbf{x},y_{<t}) \right)^2$$

Value Function Loss Minimization in RLHF

An engineer is training a language model using a reinforcement learning process. They observe that the model's value function, which is supposed to predict the expected future reward from a given state, is consistently overestimating. During a single training step, the value function's output for the current state is 5.0. The calculated target value, based on the immediate reward and the next state's value, is 3.5. The training objective is to minimize the squared error between the prediction and the target: `(target - prediction)^2`. To achieve this objective, how should the training algorithm adjust the value function's output for the current state, and why is this adjustment necessary?

Diagnosing Value Function Training

Given the following case study and the associated loss function, explain the most likely reason for the observed training behavior. Specifically, how can the value function achieve a low loss without providing a useful signal to improve the model's text generation?

Diagnosing Value Function Training Issues

During a reinforcement learning update for a language model, the value function is trained to predict future rewards. At a specific step, the value function's output for the current state is `V_current = 3.0`. The model then generates a token, for which a reward model provides a score of `r = 0.5`. The value function's output for the new state is `V_next = 4.0`. Assuming a discount factor of `γ = 0.9`, the training objective is to minimize the squared difference between `V_current` and a target 

Learn Before

Related