Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6
TL;DR
Benchmark two 30B Mixture-of-Experts models, Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B, across G5, G6, G6e, and G7 GPU instances on Amazon SageMaker AI. Compare throughput, latency, and cost-per-token, and see how G7's NVIDIA Blackwell GPUs deliver measurable price-performance gains for real-time LLM inference.
Nauti's Take
Throughput, latency and cost per token across four instance types give teams a solid basis for their own calculations, and that is the real opportunity in this run. The limit is transferability, because two 30B models measured inside an AWS setup say little about your prompts and your traffic.
Teams with a running inference bill should rebuild the benchmark, everyone else can take the direction.
Sources
