Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 → 25-29 t/s decode with the expert cache PR
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
Llama 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一位用户报告称，在配备两块3090 GPU和DDR4内存的系统上，Qwen3.8-Flash-Next模型的解码速度达到了每秒25-29个token。这一性能相比之前的每秒17个token有所提升，主要归因于使用了专家缓存PR。该设置涉及特定的llama.cpp命令，包括numactl --interleave=all build/bin/llama-server，以及-ngl 99、--moe-expert-cache 135和-t 16等参数，以实现最佳性能。
Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen.
My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV.
Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth.
Now: 25-29 t/s decode short and mid context, ~17 at 131k, prefill is still about the same. Measured this with a python coding prompt.
Now as for what I did: PR #27861, the GPU-resident LRU expert cache. Instead of parking whole expert layers in VRAM, it caches recently used experts per layer. The experts this model picks for one token are mostly the same ones it picked for the last few dozen tokens. so the hit rate is 80-85% on code and higher on prose. The trick that made it pay was giving the cache VRAM, also dropping ubatch from 2048 to 512 frees ~5 GB per GPU (compute buffers scale with ubatch), which went from 80 to 135 slots per layer at full context. The cost here is slower prefill on long prompts, short prompts are kind of unaffected.
Also, the things that did nothing or hurt on my box were: thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (verify batches re-read experts from host RAM, it only wins greedy or at deep context), and more than 2 cache uploads per step (saturates PCIe 3.0, hit rate collapses).
To replicate you don't need my fork, it's just master plus the PR:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp git fetch origin pull/27861/head:pr-27861 && git merge pr-27861 cmake -B build -DGGML_CUDA=ON && cmake --build build -j LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf \ -ngl 99 -c 261888 --parallel 1 -fa on \ -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \ --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \ --moe-expert-cache 135
Size the cache to whatever VRAM you have left after the KV and compute buffers, about 100 MB per slot per GPU on Q6. Testing UD-Q4_K_XL is the next item on my list and I'm going to revisit MTP again if it makes sense, still closely following Daniel's PR.
Doubt there's much people with a similar setup to mine out there but this helps anyone or if you have questions on approaches to try, let me know.