The model-assisted safety pipeline is an alignment framework that uses the language models themselves as tools to steer model behavior toward safe and nuanced responses. Designed to address failure modes in standard RLHF—such as generating unsafe advice or becoming overly cautious by hedging and refusing benign queries—the framework consists of two primary components: an expanded set of safety-relevant training prompts and rule-based reward models (RBRMs) that supply targeted feedback during reinforcement learning fine-tuning.

Model-Assisted Safety Pipeline

Rule-based reward models (RBRMs) are zero-shot language model classifiers that supply additional reward signals to the policy model during reinforcement learning fine-tuning. Rather than relying solely on scalar human preference models, RBRMs provide granular, rule-governed feedback to enforce safety policies, specifically rewarding the model for correctly refusing harmful requests while preventing unnecessary refusals on safe prompts.

Rule-Based Reward Models (RBRMs)

A rule-based reward model (RBRM) processes three inputs: an optional prompt, the response output generated by the policy model, and a human-written rubric structured with explicit evaluation rules (often in multiple-choice format). The classifier evaluates the response against the rubric categories—such as a refusal in the desired style, a refusal in an undesired style (e.g., rambling or evasive), disallowed content, or a safe non-refusal—allowing the system to assign tailored rewards or penalties based on the classification outcome.

Inputs and Rubric Classification Mechanism of RBRMs

Within the Reinforcement Learning from Human Feedback (RLHF) framework, the reward model is structured as a neural network. Its specific purpose is to process a pair of token sequences, consisting of an input `x` and a corresponding output `y`, and map them to a single scalar value, which represents the reward.

Function and Inputs of the RLHF Reward Model

In some applications of reinforcement learning for LLM reasoning, a reward model can be developed based on simple, predefined rules rather than being learned from data. An example of such a rule is providing a bonus or higher reward for longer, more detailed outputs to encourage the model to generate more elaborate reasoning paths.

Rule-Based Reward Models for Reasoning

University of Michigan - Ann Arbor

Ch.3 Model Alignment and Safety - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor

Impact of RLHF on Model Capability - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor

Model Calibration and Confidence Degradation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor

Model-Assisted Safety and Rule-Based Reward Models - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor

Learn Before

Related

Learn After