mistral.rs v0.9.0: up to 1.8x faster CPU decode than llama.cpp on x86 and ARM!
Llama model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
Mistral.rs v0.9.0 demonstrates up to 1.8x faster CPU decoding than llama.cpp on both x86 and ARM processors, specifically on Qwen3 4B Q4_K.…
https://preview.redd.it/nuk5rxceptbh1.png?width=1448&format=png&auto=webp&s=300344dd4c6552379e8536b81ba288be3d6dca3f
On Qwen3 4B Q4_K, mistral.rs decodes faster than llama.cpp at every context depth we measured, on x86 (Sapphire Rapids) and ARM (GB10).
We optimized mistral.rs at granular levels to achieve general speedups for all models. Additionally, our optimizations apply to CPUs of all calibers: from x86 with AVX2 or AVX512, to ARM processors with NEON.
We wanted to make sure this was a comparison in the best possible light for both engines. To ensure this, we swept various configs for mistral.rs and llama.cpp and tested at the best configuration per point for each engine.
Methodology, full tables, and repro scripts can be found here: https://github.com/EricLBuehler/mistral.rs/blob/master/releases/v0.9.0/report.md
# Mac/Linux: curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh # Windows irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex
Then, you can run any of your favorite models (Qwen 3.5/3.6, Gemma 4, LFM 2.5) directly from Hugging Face using the mistral.rs ISQ system: