Classification of LLM Fine-Tuning Approaches for Reasoning Tasks
For reasoning tasks where a Large Language Model generates a sequence of steps leading to a verifiable final answer, the fine-tuning methods can be grouped into two main categories, as identified by Uesato et al. (2022).
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.4 Alignment - Foundations of Large Language Models
Related
A development team is updating a pre-trained language model by further training it on a curated dataset of specific prompts and their desired, high-quality outputs (e.g., prompt: 'Explain gravity to a 5-year-old,' output: 'Gravity is like a big, invisible hug from the Earth...'). Why is this specific training process considered a method for model alignment?
Evaluating the Purpose of Instruction-Based Training
The process of adapting a pre-trained language model using a dataset of instructions and their corresponding desired outputs is categorized as an alignment problem because its primary goal is to enhance the model's core linguistic knowledge and predictive accuracy.
Learn After
Outcome-based Approaches for LLM Fine-Tuning
Process-based Approaches for LLM Fine-Tuning
A team is fine-tuning a large language model to solve complex, multi-step logic puzzles. They are testing two different supervisory approaches:
- Approach 1: The model generates the full sequence of reasoning steps and provides a final answer. A human evaluator then checks only if the final answer is correct. The model receives a positive signal if the answer is correct and a negative signal if it is incorrect, regardless of the reasoning steps.
- Approach 2: The model generates its rea
Recommending a Fine-Tuning Strategy for an AI Algebra Tutor
A team is fine-tuning a large language model for multi-step reasoning tasks. They are considering two general approaches for providing supervision: one that focuses only on the final answer, and one that evaluates each step of the reasoning process. Classify each of the following scenarios or characteristics by matching it to the correct supervisory approach.