As benchmarks reach a point of accuracy saturation, they are frequently retired in favor of more challenging alternatives. This practice, while understandable, may overlook critical insights.
A recent case study on CORE-Bench highlights the importance of not only measuring accuracy but also understanding the saturation effects that occur during benchmarking.
The findings suggest that the retirement of benchmarks could lead to missed opportunities for deeper research into the nuances of performance and accuracy in machine learning.