Google

A reward model can be trained on step-level annotated data to provide supervisory feedback for policy learning by treating the task as a classification problem. The model, typically a Transformer decoder with a final Softmax layer, takes the problem description ($$\mathbf{x}$$) and the preceding reasoning steps ($$\bar{\mathbf{y}}_{\le k}$$) as input at step $$k$$. It then outputs a probability distribution over a set of predefined labels, such as $$\{\text{correct}, \text{incorrect}\}$$ or $$\{\text{correct}, \text{incorrect}, \text{neutral}\}$$, to classify the current step.

Process-Based Reward Model as a Classification Task

Once a process-based reward model is trained to classify each step of a reasoning path, it can be used to evaluate the overall quality of the entire path. A straightforward evaluation method involves aggregating the step-level classifications by simply counting the total number of steps that the model identifies as 'correct'. This count serves as the overall score or reward for the reasoning path.

Scoring Reasoning Paths by Counting Correct Steps

An alternative method for evaluating a reasoning path involves using the log-probabilities generated by the step-level classification model. Instead of simply counting the number of steps deemed 'correct', this approach aggregates the log-probabilities to form the total reward for the entire path, providing a more nuanced score.

Log-Probability-Based Reward for Reasoning Paths

From a practical standpoint, applying effective supervision to long reasoning paths yields two significant benefits. It not only improves the model's overall reasoning performance but also enhances its efficiency by helping to eliminate redundant or unnecessary steps, thereby reducing the complexity of the reasoning process.

Practical Benefits of Detailed Supervision for Long Reasoning Paths

An AI team is building a supervisory model to assess each step in a multi-step reasoning process. The model receives the initial problem and all preceding steps as input, and it must output a judgment on whether the current step is 'correct' or 'incorrect'. Given this objective, which architectural component is most appropriate for the model's final layer, and why?

Given the scenario below, what specific information must be provided as input to the supervisory model to correctly classify the validity of the *second* step ('In a bowl, mix flour and sugar')? Additionally, what would be an appropriate set of output labels for this classification task?

Designing a Reward Model for a Cooking Assistant

When a process-based reward model is framed as a classification task, its primary function is to output a single, continuous score (e.g., from 0.0 to 1.0) that represents the quality of a given reasoning step.

Learn Before

Related