Build an AI-Powered A/B testing engine using Amazon Bedrock
TL;DR
AWS published a guide for building an AI-powered A/B testing engine using Amazon Bedrock, ECS, DynamoDB, and the Model Context Protocol (MCP).
Key Points
- Instead of random variant assignment, the system analyzes user context to make smarter decisions about which test variant a user sees.
- MCP acts as the bridge between the AI model and backend services – an architectural pattern gaining rapid traction.
- The stack runs containerized on ECS with experiment data and assignments stored in DynamoDB.
Nauti's Take
The approach is technically sound and the MCP pattern genuinely interesting – but this post reeks of AWS service showcase. Bedrock, ECS, DynamoDB: all proprietary, zero portability.
Anyone not already deep in the AWS ecosystem is building a golden cage here. The core idea – context-aware, AI-driven variant assignment – is worth pursuing, but it works just as well with OpenAI, LiteLLM, and a Postgres table.
The real open question is how the system handles fairness: if the AI systematically assigns 'worse' variants to certain user groups, you no longer have a testing problem – you have an ethical one.
Sources
