(NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s
Qwen 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一位开发者对NInfer（一个用于Qwen模型的C++20/CUDA推理引擎）进行了分支，以实现Qwen-3.8 27B的1M上下文。通过在两块GPU上添加张量并行处理和YaRN ×4 rope缩放，Qwen3.8-27B NVFP4模型能够在两块消费级5090显卡上运行1,048,576个token的上下文，每张卡占用27.4 GB，且无需NVLink。
Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no NVLink.
- Decode at 653k context: 119 tok/s with MTP speculative decoding, 57 without. vLLM on the same prompts: 42 tok/s.
- Why: past its native 262k window, vLLM's MTP acceptance drops to exactly zero (0 of 1,533 drafts accepted) — it keeps paying for the drafter. NInfer's acceptance stays ~55–60% out to 1M.
- Prefill is where vLLM wins: 1.2–1.3× faster. A full 1M prompt takes ~18 minutes on NInfer. That's the untuned part.
- vLLM's fp8 KV cache tops out at ~759k tokens on this hardware; NInfer's INT8 KV fits 1,048,576 in less memory.
Two GPUs are also just faster than one: 75 vs 54 tok/s at 250k, because weights and KV traffic halve per card and the ~128 cross-GPU reductions per token cost only ~0.2 ms under CUDA graphs.