Google

Intra-task generalization refers to a model's ability to perform well on new inputs for a specific task, which is indicated by a particular instruction $$c^*$$. A model is considered to generalize within this task if its average performance on these new, unseen inputs surpasses a predetermined threshold value, denoted as $$\epsilon$$.

Informal Definition of Intra-Task Generalization

Match each variable from the formal definition of intra-task generalization with its correct description. The formula is given as: $$\frac{1}{|Z|} \sum_{z' \in Z} P(c^*, z', y') > \epsilon$$

Based on the scenario below and the formal definition of intra-task generalization, has the model successfully achieved it? Justify your answer by explaining how each component of the scenario maps to the definition.

Evaluating Model Generalization in Customer Service

An instruction-tuned model is evaluated on a specific task: summarizing legal documents. The goal is to achieve intra-task generalization, which is formally defined as the average performance on a set of new inputs (Z) exceeding a predefined threshold (ε).

The evaluation uses a set of 100 new legal documents the model has never seen before. The performance threshold (ε) is set to 0.85.

- **Model A** correctly summarizes 92 of the 100 new documents.
- **Model B** correctly summarizes 81 of the 

Learn Before

Related