Last year we built megakernels. This year we deleted them: The road to Together Compiler
TL;DR
Together Compiler uses AI agents to rewrite and fuse the GPU kernels in a model’s forward pass. It beats hand-written megakernels by 28% on B200 (2,118 vs 1,649 tokens/s on Llama 3.2 1B). This is how we got from one to the other. Together Compiler uses AI agents to rewrite and fuse the GPU kernels in a model’s forward pass.
Nauti's Take
The practical test for small teams is straightforward: measure whether your inference pipeline is limited by kernel fusion, memory access, or launch overhead before adopting a compiler workflow. The 28 percent gain comes from a single company benchmark on B200 and Llama 3.2 1B, so the real question is whether the result holds for your models, batch sizes, and hardware costs.