AI Benchmarks: Opus 4.8 Performance Falls 14% Without Internet Access
TL;DR
Geeky Gadgets summarizes a Better Stack explainer on AI benchmarks: scores can be inflated by reward hacking and benchmark contamination, making models look more capable than they are. The standout claim: Opus 4.8 reportedly loses 14% performance under stricter offline conditions, when web access and historical code repositories are removed. Qwen 2.5 is also cited: on SST-2, its score allegedly drops from 90% to 30-40% after contamination is accounted for. The piece is secondary and fairly PR-heavy.
Nauti's Take
The 14% drop is interesting, but the deeper issue is the habit of mixing lab performance with work performance. A model with web access is not automatically worse in real workflows, but the benchmark has to say clearly what it measures.
Offline tests, tool-use tests, and agent tests should be reported separately. Otherwise the leaderboard is mostly ranking makeup.
Briefingshow
Benchmarks shape buying decisions, model rankings, and product marketing. If a model has seen the test data or can retrieve ready-made solutions from the web, the score measures access and memory as much as reasoning. Teams should treat leaderboards as a signal, not a substitute for their own workflow tests with real files, policies, and tool limits.
Video
Sources