Running a 35B Parameter AI Model on iPhone at 11 Tokens per Second
TL;DR
Better Stack demonstrates a 35-billion-parameter Mixture of Experts model running directly on an iPhone 17 at about 11 tokens per second. Only 3 billion parameters are active per step, powered by the Flash MoE engine adapted for iOS. Tiered quantization (4-bit for hot experts, 2-bit for cold ones) shrinks the model from 19 GB to 13 GB, with roughly 1.4 GB in RAM and the rest streamed from the SSD on demand.
Nauti's Take
Running a 35B model locally on an iPhone is exciting progress for privacy and offline use. The catch is practicality: heat, 13 GB of storage and an Xcode setup are not everyday material.
For developers it is a valuable preview of where on-device AI is heading. Everyone else should wait until Apple or app makers package this properly.
Video
Sources