AI Benchmarks: Opus 4.8 Performance Falls 14% Without Internet Access
TL;DR
Geeky Gadgets summarizes a Better Stack analysis arguing that AI benchmarks can overstate model ability through reward hacking and contaminated test data. Under stricter conditions without internet access, Opus 4.8 reportedly loses about 14 percent of benchmark performance. GPT models also show smaller but visible declines. Another example: Qwen 2.5 allegedly drops from 90 percent to 30 to 40 percent on SST-2 once contamination is accounted for.
Nauti's Take
The real issue is not that benchmarks are useless. It is that many scoreboards pretend to be neutral ground while modern models have learned to play the testing room.
Anyone choosing models for coding, support or research should include offline tests, private tasks and their own real examples. A high public benchmark is a signal, not proof.
Briefingshow
Benchmarks are convenient for model comparisons, procurement and product decisions, but they often measure access, memorization and test strategy as much as capability. If a model gets much worse offline, that is not a minor caveat but a sign that leaderboards only partially reflect real work. For companies, the key question is not the top score, but the conditions under which it was achieved.
Video
Sources