---
title: "AI benchmarks are broken. Here’s what we need instead."
slug: "ai-benchmarks-are-broken-heres-what-we-need-instead"
date: 2026-03-31
category: tech-pub
tags: []
language: en
sources_count: 1
featured: false
publisher: AInauten News
url: https://news.ainauten.com/en/story/ai-benchmarks-are-broken-heres-what-we-need-instead
---

# AI benchmarks are broken. Here’s what we need instead.

**Published**: 2026-03-31 | **Category**: tech-pub | **Sources**: 1

---

## TL;DR

- AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.

---

## Summary

- AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.
- The 'AI vs. human' framing is catchy but misleading: it does not capture how AI performs in real, complex work environments.
- Current benchmarks get saturated fast – once a model tops a leaderboard, a new test is needed, without reflecting genuine capability gains.
- Researchers are calling for evaluation frameworks that measure system performance in real-world workflows, not point-in-time tasks against a human baseline.

---

## Why it matters

AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.

---

## Key Points

- AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.
- The 'AI vs. human' framing is catchy but misleading: it does not capture how AI performs in real, complex work environments.
- Current benchmarks get saturated fast – once a model tops a leaderboard, a new test is needed, without reflecting genuine capability gains.
- Researchers are calling for evaluation frameworks that measure system performance in real-world workflows, not point-in-time tasks against a human baseline.

---

## Nauti's Take

It is an open secret in the AI industry: benchmarks get optimized, not capabilities. Models are trained or fine-tuned toward test sets until the numbers shine – what follows in practice is often disillusionment. The 'beats humans' narrative is marketing, not measurement. What is genuinely missing are evaluations that assess whether AI systems reliably perform in specific professional contexts over weeks – not whether a model passes an SAT on a Tuesday.

---


## FAQ

**Q:** What is AI benchmarks are broken. Here’s what we need instead. about?

**A:** - AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.

**Q:** Why does it matter?

**A:** AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.

**Q:** What are the key takeaways?

**A:** AI models have long been evaluated by whether they beat individual humans on isolated tasks – chess, math, coding, essay writing.. The 'AI vs. human' framing is catchy but misleading: it does not capture how AI performs in real, complex work environments.. Current benchmarks get saturated fast – once a model tops a leaderboard, a new test is needed, without reflecting genuine capability gains.

---

## Related Topics

- —

---

## Sources

- [AI benchmarks are broken. Here’s what we need instead.](https://www.technologyreview.com/2026/03/31/1134833/ai-benchmarks-are-broken-heres-what-we-need-instead/) - MIT Technology Review

---

## About This Article

This article is a synthesis of 1 sources, curated and summarized by AInauten News. We aggregate AI news from trusted sources and provide bilingual (German/English) coverage.

**Publisher**: [AInauten](https://www.ainauten.com) | **Site**: [news.ainauten.com](https://news.ainauten.com)

---

*Last Updated: 2026-04-01*