9 / 2338

Making open models work for long-horizon agentic workloads

TL;DR

Every agent turn resends the full session. Together AI and Ollama break down how to protect the prefix cache, from the harness to the GPU. Every agent turn resends the full session. Together AI and Ollama break down how to protect the prefix cache, from the harness to the GPU.

Nauti's Take

Before deploying an open model for long-running agents, verify that your harness keeps the shared context stable and that the serving stack preserves prefix caching across turns. The Together AI post is a vendor perspective without independent latency, memory, or cost data.

Measure those three variables with your own workflow before changing the infrastructure.

Sources