PrismML released Bonsai 27B, a 27-billion-parameter language model that can run on mobile devices through advanced quantization techniques, reducing its size from ~50GB to ~4GB. The model achieves competitive performance while enabling on-device inference. This represents a significant step for on-device AI, bringing large-scale model capabilities to smartphones and edge devices without cloud dependency. It could enable new privacy-preserving and offline applications, while sparking interest from major players like Apple. The model is likely based on the Qwen 2.5 architecture (as per community hints) and uses ternary quantization (BitNet-like) to achieve extreme compression. However, community benchmarks indicate tool-calling performance is notably degraded compared to similar-size models like Gemma 4 12B.
Background
Quantization is a model compression technique that reduces the precision of weights and activations, shrinking model size and enabling faster inference on limited hardware. On-device AI allows models to run locally without cloud calls, improving privacy and latency. Bonsai 27B pushes the frontier of what size model can be feasibly deployed on a phone.
References
Discussion
Commenters are excited about the compression but cautious about trade-offs: one user noted that the demo recipe had wrong macronutrients, and tool-calling performance is a known weakness. Another compared it favorably to Gemma 4 12B QAT but questioned if the quantization loss is minimal. Apple's reported talks add credibility to the approach.