AI Benchmarks: Opus 4.8 Performance Falls 14% Without Internet Access
TL;DR
Geeky Gadgets summarizes a Better Stack analysis arguing that AI benchmarks can be inflated by reward hacking and contaminated test data. Opus 4.8 reportedly loses 14 percent performance under stricter offline conditions, where web access and public repositories are no longer available as shortcuts. The article also cites Qwen 2.5 on SST-2: after accounting for contamination, performance allegedly drops from 90 percent to 30 to 40 percent.
Nauti's Take
The piece is useful, but it is clearly secondary reporting: it relays a Better Stack argument and does not present a full primary study in the article itself. The core point still lands.
Buying models in 2026 from leaderboard numbers alone ignores tool access, training leakage, and test-set familiarity. The better benchmark is your messiest real task, run with the same permissions, data, and limits the model will face in daily work.
Briefingshow
Leaderboards only help if they measure what users will actually get in production. If a model solves coding tasks through GitHub lookups or leaked test patterns, buyers are not seeing reasoning, they are seeing shortcut behavior. Teams should pair public benchmark scores with offline checks, private test cases, and workflow trials before choosing a model.
Video
Sources