Google

The primary goal of Supervised Fine-Tuning (SFT) is to adapt a model that already has pre-trained parameters, denoted as $$\hat{\theta}$$. The adaptation process involves adjusting these parameters to maximize the conditional probability of generating the desired output sequence, $$y$$, given a specific input sequence, $$x$$. This objective, expressed as maximizing $$\mathrm{Pr}(y|x)$$, aligns the model's behavior with the provided instruction-response examples.

Objective of Supervised Fine-Tuning

In Supervised Fine-Tuning (SFT), the training objective is to maximize the probability of the model generating a "gold-standard" or ground-truth output ($y$) given a specific input ($x$). This is achieved through Maximum Likelihood Estimation (MLE), where the model's parameters are adjusted to make its predicted token distributions align as closely as possible with the one-hot encoded distributions of the correct response. The formal objective is to maximize the conditional probability: $max \ Pr(y|x)$.

Maximum Likelihood Estimation (MLE) as the Objective for Supervised Fine-Tuning

An engineer is fine-tuning a language model on a dataset of instruction-response pairs. They state their goal is to adjust the model's parameters to maximize the probability of the input instruction, given the model's generated response. Identify the fundamental error in this stated objective and describe the correct probabilistic objective for this task.

Correcting a Flawed Fine-Tuning Objective

Based on the core principle of this training method, what is the team's primary objective? Explain how this objective relates the model's parameters to the provided dataset.

Objective for a Specialized Math Tutor

In supervised fine-tuning (SFT), the goal is to adjust pre-trained model parameters to maximize the conditional probability of the target output sequence $$\mathbf{y}$$ given the input sequence $$\mathbf{x}$$. Given pre-trained parameters $$\hat{\theta}$$ and a dataset $$\mathcal{D}$$ of input-output pairs, the objective is to find the optimized parameters $$\tilde{\theta}$$ by maximizing the sum of conditional log-probabilities: $$\tilde{\theta} = \arg\max_{\hat{\theta}^+} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}} \log \mathrm{Pr}_{\hat{\theta}^+}(\mathbf{y}|\mathbf{x})$$. This formulation highlights that optimization starts from the pre-trained weights rather than from random initialization.

Mathematical Formulation of the Supervised Fine-Tuning Objective

A fundamental difference exists between standard language modeling and supervised fine-tuning objectives. Standard language modeling minimizes the loss over all tokens of a concatenated input-output sequence $$\mathrm{seq}_{\mathbf{x},\mathbf{y}} = [\mathbf{x},\mathbf{y}]$$, optimizing the joint log-probability $$\log \mathrm{Pr}_{\theta}(\mathbf{x},\mathbf{y})$$. In contrast, fine-tuning focuses on the conditional log-probability of the output. By applying the chain rule, the joint sequence probability is decomposed into the probability of the input and the conditional probability of the output: $$\log \mathrm{Pr}_{\theta}(\mathrm{seq}_{\mathbf{x},\mathbf{y}}) = \log \mathrm{Pr}_{\theta}(\mathbf{x}) + \log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$$. In the fine-tuning context, the loss computation over the input $$\mathbf{x}$$ is set to $${}0$$, meaning the loss is computed exclusively for the output tokens $$\mathbf{y}$$.

Conditional vs. Joint Probability Objectives in Language Modeling

A development team is fine-tuning a pre-trained language model using a curated dataset of customer support inquiries (inputs) and their corresponding ideal, human-written responses (outputs). The aim is to create a specialized chatbot that reliably provides answers in the same helpful and accurate style as the examples. From a probabilistic perspective, which statement best describes the fundamental objective of this training process?

Learn Before

Related