Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 → 25-29 t/s decode with the expert cache PR

Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen.

My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV.

Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth.

Now: 25-29 t/s decode short and mid context, ~17 at 131k, prefill is still about the same. Measured this with a python coding prompt.

Now as for what I did: PR #27861, the GPU-resident LRU expert cache. Instead of parking whole expert layers in VRAM, it caches recently used experts per layer. The experts this model picks for one token are mostly the same ones it picked for the last few dozen tokens. so the hit rate is 80-85% on code and higher on prose. The trick that made it pay was giving the cache VRAM, also dropping ubatch from 2048 to 512 frees ~5 GB per GPU (compute buffers scale with ubatch), which went from 80 to 135 slots per layer at full context. The cost here is slower prefill on long prompts, short prompts are kind of unaffected.

Also, the things that did nothing or hurt on my box were: thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (verify batches re-read experts from host RAM, it only wins greedy or at deep context), and more than 2 cache uploads per step (saturates PCIe 3.0, hit rate collapses).

To replicate you don't need my fork, it's just master plus the PR:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp git fetch origin pull/27861/head:pr-27861 && git merge pr-27861 cmake -B build -DGGML_CUDA=ON && cmake --build build -j LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf \ -ngl 99 -c 261888 --parallel 1 -fa on \ -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \ --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \ --moe-expert-cache 135

Size the cache to whatever VRAM you have left after the KV and compute buffers, about 100 MB per slot per GPU on Q6. Testing UD-Q4_K_XL is the next item on my list and I'm going to revisit MTP again if it makes sense, still closely following Daniel's PR.

Doubt there's much people with a similar setup to mine out there but this helps anyone or if you have questions on approaches to try, let me know.