How an $8 ESP32 S3 Microcontroller Runs a 28.9M Parameter Local LLM
TL;DR
An 8 dollar ESP32 S3 microcontroller has just 0.5 MB of fast SRAM and 8 MB of slower PSRAM, and is normally used for lightweight jobs such as managing Wi-Fi. The Stack shows how a language model with 28.9 million parameters can still run on it. The result comes from hardware-aware optimisations and a set of creative engineering tradeoffs rather than any single trick.
Nauti's Take
Nice progress for edge AI: a 28.9 million parameter model on an eight dollar chip shows how far hardware aware optimization now goes. The limit is model size, because at that parameter count you are looking at narrow tasks, not assistants.
Exciting for makers and hardware teams with a well defined use case, while anything language heavy still needs the cloud.
Video
Sources