Autoscaling endpoints for LLM inference
TL;DR
GPU utilization can read healthy while the request queue backs up, and a new replica can take minutes to warm. The piece covers how to choose autoscaling metrics for LLM inference that reflect real latency rather than raw utilization, how to tune scale-up and scale-down windows, and how to budget for cold starts on dedicated inference endpoints.
Nauti's Take
The core point is genuinely useful: GPU utilisation alone is a misleading autoscaling signal, and combining queue length, latency, and in-flight tokens saves real money on LLM inference. The catch is the source, since Together AI sells exactly this inference, and the multi-minute warm-up applies to dedicated serving rather than every setup.
Small teams get more from running a load profile with realistic prompt lengths and burst traffic before setting scale-up windows on someone else's numbers.
Sources