Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload
TL;DR
Comparing models on dollars per million tokens misses what production workloads actually pay for: outcomes. This post shares an open-source benchmarking harness that measures cost per correct answer, agent trajectory cost, and rubric-graded deliverable quality across OpenAI models on Amazon Bedrock.
Nauti's Take
The harness finally measures the number that decides production budgets: cost per correct answer, plus the full cost of an agent trajectory. That is a real advantage for teams running agents at scale who have been guessing so far.
The limitation is scope. Results cover OpenAI models on Bedrock and the harness tasks only, and rubric grading stays a judgement call.
Teams should rerun it on their own workloads before trusting the ranking.
Sources
