In test set contamination analysis, non-contaminated questions represent leaked items, while contaminated questions represent unseen items.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.2 Model Scaling and Capability Evaluation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Test Set Contamination Analysis - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Ch.1 Foundation Model Capabilities and Benchmarking - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Benchmark Contamination Analysis - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Related
How is performance degradation calculated when evaluating test set contamination in language models?
In the context of test set contamination analysis, what does the performance degradation metric quantify?
A language model achieves a score of 62% on the non-contaminated questions of an evaluation benchmark and 88% on the contaminated questions of the same benchmark.
- Calculate the performance degradation metric for this evaluation.
- Interpret what this specific result indicates regarding the model's reported capability metrics.
Contamination as a Non-Substantive Confounder in GPT-4 Exam Evaluation