AI Benchmarks: Opus 4.8 Performance Falls 14% Without Internet Access
TL;DR
Geeky Gadgets summarizes a Better Stack analysis arguing that AI benchmarks can overstate model capability through reward hacking and benchmark contamination. Under stricter tests without internet access and old code repositories, Opus 4.8 reportedly drops 14 percent; GPT models also decline, but less sharply according to the article. The article cites Qwen 2.5 on SST-2 as a contamination example: after accounting for exposure, performance allegedly falls from about 90 percent to 30 to 40 percent.
Nauti's Take
Treat public benchmark scores for coding and workflow agents as a shortlist signal, not a buying argument. The first thing a small team should verify is the same task in its own environment, without web access, using its own repos and clear pass criteria.
The source base is thin, so the exact Opus number remains unverified, but the testing rule is still useful.
Briefingshow
Benchmark scores shape which models teams test, buy, and wire into workflows. If a model is mainly good at recognizing known tasks or retrieving public solutions, its real working ability is overstated. For agents, coding assistants, and tool-using models, the test setup matters more now: online or offline, with tools or without tools, public tasks or private test sets.
Video
Sources