---
title: "Show HN: ACE – A dynamic benchmark measuring the cost to break AI agents"
slug: "show-hn-ace-a-dynamic-benchmark-measuring-the-cost-to-break-ai-agents"
date: 2026-04-05
category: community
tags: [anthropic, agents]
language: en
sources_count: 1
featured: false
publisher: AInauten News
url: https://news.ainauten.com/en/story/show-hn-ace-a-dynamic-benchmark-measuring-the-cost-to-break-ai-agents
---

# Show HN: ACE – A dynamic benchmark measuring the cost to break AI agents

**Published**: 2026-04-05 | **Category**: community | **Sources**: 1

---

## TL;DR

- The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.

---

## Summary

- The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.
- Six budget-tier models were tested under identical agent configurations: Gemini Flash-Lite, DeepSeek v3.2, Mistral Small 4, Grok 4.1 Fast, GPT-5.4 Nano, and Claude Haiku 4.5.
- Claude Haiku 4.5 was an order of magnitude harder to break: mean adversarial cost of $10.21 vs. $1.15 for the next-best model (GPT-5.4 Nano). The remaining four all fell below $1.
- ACE enables game-theoretic reasoning – at what cost does an attack become economically rational? That reframes AI security evaluation fundamentally.

---

## Why it matters

The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.

---

## Key Points

- The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.
- Six budget-tier models were tested under identical agent configurations: Gemini Flash-Lite, DeepSeek v3.2, Mistral Small 4, Grok 4.1 Fast, GPT-5.4 Nano, and Claude Haiku 4.5.
- Claude Haiku 4.5 was an order of magnitude harder to break: mean adversarial cost of $10.21 vs. $1.15 for the next-best model (GPT-5.4 Nano). The remaining four all fell below $1.
- ACE enables game-theoretic reasoning – at what cost does an attack become economically rational? That reframes AI security evaluation fundamentally.

---

## Nauti's Take

A benchmark that prices security in dollars is not a gimmick – it speaks the language that budget holders actually understand. The fact that four of six tested models can be broken for under a dollar should alarm anyone running agents with real permissions and real data access. The Haiku 4.5 outlier is fascinating, but caution is warranted: six models, one setup, early methodology – this is a promising first swing, not a definitive verdict. What the community needs now is independent replication and an honest debate about whether 'adversarial cost' truly holds up across different attack strategies.

---


## FAQ

**Q:** What is Show HN about?

**A:** - The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.

**Q:** Why does it matter?

**A:** The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.

**Q:** What are the key takeaways?

**A:** The team built 'Adversarial Cost to Exploit' (ACE), a benchmark quantifying how many tokens – expressed in dollars – an autonomous adversary must spend to breach an LLM agent, replacing binary pass/fail metrics.. Six budget-tier models were tested under identical agent configurations: Gemini Flash-Lite, DeepSeek v3.2, Mistral Small 4, Grok 4.1 Fast, GPT-5.4 Nano, and Claude Haiku 4.5.. Claude Haiku 4.5 was an order of magnitude harder to break: mean adversarial cost of $10.21 vs. $1.15 for the next-best model (GPT-5.4 Nano). The remaining four all fell below $1.

---

## Related Topics

- [anthropic](https://news.ainauten.com/en/tag/anthropic)
- [agents](https://news.ainauten.com/en/tag/agents)

---

## Sources

- [Show HN: ACE – A dynamic benchmark measuring the cost to break AI agents](https://fabraix.com/blog/adversarial-cost-to-exploit) - Hacker News AI

---

## About This Article

This article is a synthesis of 1 sources, curated and summarized by AInauten News. We aggregate AI news from trusted sources and provide bilingual (German/English) coverage.

**Publisher**: [AInauten](https://www.ainauten.com) | **Site**: [news.ainauten.com](https://news.ainauten.com)

---

*Last Updated: 2026-04-06*