Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips
TL;DR
Exa's neural search engine is backed by an in-house vector database, so the unit economics of embedding tens of billions of web documents drive the business. This post shares how Exa optimized embedding inference on AWS Neuron hardware (Inferentia2 and Trainium2) with custom NKI kernels, reaching a 47% gain in performance and cost per million documents.
Nauti's Take
The real story is the unit economics: indexing billions of documents turns embedding inference into an infrastructure problem. Smaller teams should first benchmark cost per million documents across their current model and hardware options before building a similar optimization layer.