Today I hit 181 toks/s (aggregate) on Qwen3.8-Flash-Next on 2x DGX Sparks
Heat trend
The percentage is based on available heat signal, not comment count or independent people.
NVIDIA model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A developer achieved an aggregate throughput of 181 tok/s on Qwen3.8-Flash-Next using two NVIDIA DGX Spark nodes. This was accomplished with a 512K context and 2.8M KV cache, utilizing an NVFP4 quant model. Key optimizations included offloading a 47.7 GiB n-gram embedding table to NVMe with madvise(MADV_RANDOM) and 64 gather threads, which reduced memory usage from 65 GiB to 41 GiB per node. The vLLM configuration also played a crucial role, with --kv-cache-memory and --enable-prefix-caching being significant contributors.
Today I managed some pretty crazy numbers: 181 tok/s aggregate on 2× DGX Spark on Qwen3.8-Flash-Next at 512Kcontext (2.8M kvc)
I hit 181 tok/s aggregate today across a multi-agent fleet on a 2-node DGX Spark cluster. Single-stream decode is 30–50 tok/s — the 181 is total throughput with ~9 concurrent agent sessions sharing the engine. I actually peaked to 195 while writing this. Quick rundown of how it's served:
- 2× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory each, 20-core ARM)
- Nodes linked with a direct ConnectX-7 cable — NCCL over RDMA (RoCE, 200 Gb), TP=2 across both boxes
- Verify Using network IB in the NCCL log — the TCP fallback is silent and costs you half your speed
- Qwen3.8-Flash-Next, RadixArk NVFP4 quant (4-bit routed experts, FP8 n-gram table)
- Hybrid arch: 3/4 linear attention + 1/4 sparse full attention, 512-expert MoE, MTP speculative decoding k=3 (~40% acceptance)
- Native 262K context stretched to 512K with YaRN factor 2.0 — needle-verified at 487K depth
- The model carries a 320M-row n-gram embedding table (47.7 GiB in FP8) that's read on every token — but each token only touches 16 rows (~2.5 KB)
- We mmap it straight off NVMe instead of loading it: weights per node dropped 65 → 41 GiB
- Two things made it fast: madvise(MADV_RANDOM) on the mapping (hash-scattered lookups + kernel readahead = 30× read amplification — one 405K prefill read 603 GB from disk before the fix, 19 GB after) and 64 gather threads (the wall was fault latency serialization, not disk bandwidth)
- Freed memory went into KV: pool is now 2.89M tokens (5.5× full contexts) at a 40.6 GiB pin
- --kv-cache-memory 40600000000 — pin the pool explicitly; heads-up, a manual pin ignores --gpu-memory-utilization, size it from measured free RAM
- --max-num-batched-tokens 8192, --long-prefill-token-threshold 4096 — protects decode latency during cold prefills
- --enforce-eager — CUDA graphs crash this build on GB10/SM121 (torch.compile AOT dies on rank 1)
- --enable-prefix-caching — 99% hit rate with agent traffic, this is the single biggest real-world win
- Small scheduler patch: cap concurrent cold long prefills (admission gate) so N agents prefilling at once can't balloon host RAM — on unified memory, prefill transients eat the same pool as your weights
- llama-swap in front (one model resident at a time, API-key auth, model swap on request), nginx TLS for outside access
- earlyoom with absolute floors as the only watchdog — on a unified-memory box, low free RAM with a big model resident is normal , percentage-based OOM killers will shoot a healthy model