Google

The formula for calculating the Performance Gap Recovered (PGR) is: $$ \text{PGR} = \max\left\{0, \frac{\text{P}_{\text{weak}\to\text{strong}} - \text{P}_{\text{weak}}}{\text{P}_{\text{ceiling}} - \text{P}_{\text{weak}}}\right\} $$ In this equation, $$ \text{P}_{\text{weak}} $$ represents the baseline performance of the weak model, $$ \text{P}_{\text{weak}\to\text{strong}} $$ is the performance of the strong model after being supervised by the weak model, and $$ \text{P}_{\text{ceiling}} $$ is the strong model's maximum potential performance.

Formula for Performance Gap Recovered (PGR)

A research team is developing a new model. Their initial, 'weak' model achieves an accuracy of 50% on a specific classification task. They have a more powerful, 'strong' model architecture which, if trained on a perfect dataset, could theoretically reach a 'ceiling' accuracy of 90%. To save on labeling costs, they use the weak model to supervise the training of the strong model. After this process, the newly trained strong model achieves an accuracy of 80%. Based on this scenario, calculate the Performance Gap Recovered (PGR).

PGR Calculation Scenario

In the formula used to calculate the extent to which a strong model's performance improves through supervision by a weak model, the denominator is calculated as the difference between the strong model's maximum potential performance and the weak model's baseline performance. What does this denominator represent in the context of the overall calculation, and why is it a crucial component of the formula?

Interpreting the PGR Formula's Denominator

In an experiment, a researcher observes that the performance of a strong model after being supervised by a weak one (P_weak→strong) is actually lower than the weak model's initial baseline performance (P_weak). Assuming the strong model's maximum potential performance (P_ceiling) is greater than the weak model's baseline, what is the resulting Performance Gap Recovered (PGR) and what does it signify?

Learn Before

Related