Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
TL;DR
Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that extends the cache into a shared, distributed NVMe pool with Curvine, so replicas reuse cache at near-local-disk speeds on cost-efficient instances.
Nauti's Take
Start by testing the design against your own prompt patterns and a realistic number of concurrent replicas. Measure time-to-first-token, cache hit rate, NVMe cost, and invalidation behavior before treating the architecture as a cost-saving default.
The AWS post demonstrates the pattern, but it does not yet provide independent evidence across different workloads.
Sources
11.8.26
