(NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Qwen model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A developer forked NInfer, a C++20/CUDA inference engine for Qwen models, to enable a 1M context for Qwen-3.8 27B. This was achieved by adding tensor-parallel processing across two GPUs and YaRN ×4 rope scaling. This setup allows the Qwen3.8-27B NVFP4 model to run a 1,048,576-token context on two consumer 5090 GPUs, utilizing 27.4 GB per card without NVLink.
Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no NVLink.
- Decode at 653k context: 119 tok/s with MTP speculative decoding, 57 without. vLLM on the same prompts: 42 tok/s.
- Why: past its native 262k window, vLLM's MTP acceptance drops to exactly zero (0 of 1,533 drafts accepted) — it keeps paying for the drafter. NInfer's acceptance stays ~55–60% out to 1M.
- Prefill is where vLLM wins: 1.2–1.3× faster. A full 1M prompt takes ~18 minutes on NInfer. That's the untuned part.
- vLLM's fp8 KV cache tops out at ~759k tokens on this hardware; NInfer's INT8 KV fits 1,048,576 in less memory.
Two GPUs are also just faster than one: 75 vs 54 tok/s at 250k, because weights and KV traffic halve per card and the ~128 cross-GPU reductions per token cost only ~0.2 ms under CUDA graphs.