Match each stage, metric value, or procedure to its role in large language model calibration on the MMLU benchmark.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.3 Model Alignment and Safety - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Model Calibration and Confidence Degradation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Related
Order the progression of model calibration and confidence behavior from the pre-trained state through post-training alignment.
Explain how the post-training alignment impacted the model's calibration and what the increase in ECE signifies regarding the model's confidence versus its accuracy.
In pre-trained GPT-4, what relationship is observed between the model's predicted probabilities (logprobs) across multiple-choice options and its actual task accuracy on benchmarks like MMLU?
Based on benchmark evaluations on MMLU, how does post-training alignment quantitatively alter GPT-4's Expected Calibration Error (ECE)?