Google

In the context of the reward model loss function, the parameters denoted by ϕ encompass all trainable elements of the model. Specifically, this includes the parameters of the underlying Transformer decoder architecture as well as the weights of the final linear mapping matrix, Wr, which transforms the model's internal representations into a scalar reward score.

Composition of Reward Model Parameters (ϕ)

A reward model is typically constructed by adapting a pre-trained language model. Identify and briefly describe the two main categories of trainable parameters that are collectively represented by ϕ in this context.

Components of Reward Model Parameters

In the context of training a reward model, the parameter set ϕ, which is optimized to minimize the loss function, consists solely of the weights of the final linear layer responsible for mapping the model's internal representations to a scalar reward score.

A reward model is constructed by taking a large, pre-trained language model and adding a new linear layer on top to output a single scalar value. To train this model efficiently, an engineer freezes the weights of the pre-trained language model and only updates the weights of the new linear layer. How does this training strategy relate to the complete set of the reward model's parameters, denoted as ϕ?

Learn Before

Related