9 / 2338

Last year we built megakernels. This year we deleted them: The road to Together Compiler

TL;DR

Together Compiler uses AI agents to rewrite and fuse the GPU kernels in a model’s forward pass. It beats hand-written megakernels by 28% on B200 (2,118 vs 1,649 tokens/s on Llama 3.2 1B). This is how we got from one to the other. Together Compiler uses AI agents to rewrite and fuse the GPU kernels in a model’s forward pass.

Nauti's Take

The practical test for small teams is straightforward: measure whether your inference pipeline is limited by kernel fusion, memory access, or launch overhead before adopting a compiler workflow. The 28 percent gain comes from a single company benchmark on B200 and Llama 3.2 1B, so the real question is whether the result holds for your models, batch sizes, and hardware costs.

Sources