Importance 2 of 5Covered by 1 source
Apple paper proposes a theory of evaluability for foundation models
Apple researchers formalize when models can be reliably evaluated without assuming the test set was held out from training. The work addresses accuracy, safety, calibration and fairness measures for large foundation models.
Why it matters
Contamination of test sets undermines benchmark claims companies rely on when choosing models.