Evaluating Deep Agents using LangSmith on AWS
TL;DR
This guide combines LangChain's work on evaluating deep agents with Anthropic's eval playbook into a hands-on workflow. You'll learn five evaluation patterns, build offline evals with pytest and LangSmith, and configure online monitoring for production. A text-to-SQL deep agent on Amazon Bedrock serves as the running example from development through deployment.
Nauti's Take
A practical guide to deep agent evaluation is a real opportunity to make agent quality measurable instead of relying on demos. The risk: such evals lock you early into specific stacks (Bedrock, LangSmith), which raises switching costs in a fast-moving market.
Teams should adopt the five patterns but document the tool dependencies explicitly.
Sources
