Qwen3.8-Flash-Next (95.5 GiB) on a 64GB Mac at ~27 tok/s, checkpoint + fork
A user successfully ran the Qwen3.8-Flash-Next model, which is 95.5 GiB, on a 64 GB Mac at approximately 27 tokens/second. This was achieved by utilizing expert streaming, a technique developed by mihailescu2m, where routed experts are stored on the SSD and only loaded when needed. Performance was further optimized through tuning, with a depth 3 configuration and a 0.3 confidence floor outperforming depth 4. Additionally, gather-based sparse attention (#28213) and Metal MoE fusion (#28948) contributed significant improvements.
Unlike other reports focusing on raw performance, this account details specific tuning parameters and open pull requests that enabled running a 95.5 GiB model on a 64 GB Mac.
时间与来源
时间显示为 UTC
显示时区：UTC
本地时区尚不可用，暂时显示 UTC。
发布当时偏移：UTC+02026年9月18日 20:15 UTC
收录当时偏移：UTC+02026年9月19日 13:00 UTC
- 发布
- 2026年9月18日 20:15
- 收录
- 2026年9月19日 13:00
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
档位是按信源手工设定的编辑判断，不是逐条打分。
I've been running Qwen3.8-Flash-Next as my main local coding model from past few weeks. The file is 95.5 GiB and my Mac has 64 GB. It works because the routed experts stay on SSD and only get read when a token actually routes to them.
Finally cleaned it up enough to publish:
- model: https://huggingface.co/nitinpanj/qwen38-flash-next-v3 .
- fork: https://github.com/npanj/llama.cpp
This won't run on stock llama.cpp. Upstream has the architecture but not the expert streaming flag, so it just tries to load all 95.5 GiB and falls over.
Speeds on my M5 Pro (64 GB):
- prompt processing u/4k ~367 tok/s
- generation, draft head on ~27.6 tok/s
- generation, draft head off 18-18.6 tok/s
- at 29k context, real chat use ~20.6 tok/s
What made prefill fast: the per-layer embedding table is 26.8 GiB with 90-byte rows, and under mmap you eat a page fault per gather. Switching to direct file reads (pwilkin's #29030, still open upstream) took 512-token prompts from 181 to 401 tok/s, and 8k prompts from 274 to 451.
What made decode fast: mostly the draft head. Small model guesses 3 tokens, big model verifies them in one pass. Same machine, same afternoon, 18 -> 27.58 tok/s.
Tuning it mattered nearly as much as having it: depth 3 with a 0.3 confidence floor beat depth 4 with no floor by 10% over a 22-arm sweep. Depth 4 lost every time I measured it, which surprised me since the fork hardcodes 4. On top of that, gather-based sparse attention (#28213) is worth +19% at 62k context and +50% at 130k, and Metal MoE fusion (#28948) another 5-9%.
The checkpoint is bartowski's Q4_0 with a Q8_0 output layer, plus unsloth's UD-IQ4_XS spliced into the five tensor groups that stay resident. Paired 40-chunk perplexity went 5.2777 -> 4.3148, better on all 40 chunks, for 1.69 GiB more on disk.
Setup is five steps and about an hour, mostly downloading. Commands are in the repo readme. Two things that cost me time: Keep the expert cache and the Metal wired limit in sync. On 64 GB, 36 GiB of cache is the practical ceiling. I tried 38 and decod collapsed to 3.7 tok/s once the server's own heap started swapping. Only tested on 64GB Apple Silicon, internal SSD. Nothing tried on CUDA or CPU.
Credit where it's due: expert streaming is mihailescu2m's work and it's what makes the whole approach possible. Base model is Qwen's, quants from bartowski and unsloth. The six still-open PRs I'm carrying are listed with authors in the readme.