Google

The formula $$r(\mathbf{x}, y) = \mathbf{h}_{\text{last}} \mathbf{W}_r$$ defines a reward function where the reward $$r$$ for a given prompt $$\mathbf{x}$$ and generated output $$y$$ is calculated as a linear function of the final hidden state, $$\mathbf{h}_{\text{last}}$$, of the language model that produced $$y$$. Here, $$\mathbf{h}_{\text{last}}$$ is the vector representation of the last token in the output sequence, and $$\mathbf{W}_r$$ is a learned weight matrix or vector that transforms this hidden state into a scalar reward value.

Reward Function as a Linear Transformation of the Last Hidden State

Based on the provided formula for the reward model, explain the most likely reason why the contradictory response described in the case study received a high reward.

Reward Model Behavior Analysis

A language model's reward function is defined by the equation $r = \mathbf{h}_{\text{last}} \mathbf{W}_r$, where $r$ is the scalar reward, $\mathbf{h}_{\text{last}}$ is the vector representation of the final token in the generated output, and $\mathbf{W}_r$ is a learned weight matrix. Based on this formula, explain one significant advantage and one significant disadvantage of this approach for evaluating the quality of a generated text.

Evaluating a Reward Calculation Method

A reward model for a generative text model calculates a quality score for a given output using the formula $r = \mathbf{h}_{\text{last}} \mathbf{W}_r$. In this formula, $\mathbf{h}_{\text{last}}$ is the vector representation of the final token in the generated text, and $\mathbf{W}_r$ is a learned weight matrix that transforms this vector into a scalar score, $r$. What is a primary conceptual limitation of this specific reward calculation method, especially when evaluating lengthy and complex te

Learn Before

Related