NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
这条记录涉及生成能力或端侧推理进展，适合跟踪模型效率、部署门槛和应用机会。
NInfer 现已为 Qwen3.8-27B 提供 Day-0 支持，在单张 RTX 5090 上通过推测解码可达到约 200 tok/s 的生成速度。NInfer 引擎近期进行了多项改进，包括支持多达 8 个并发请求，并共享分页 KV 缓存池，每个请求仍可使用完整的上下文长度。…
Qwen3.8-27B is finally here, and NInfer already has Day-0 support!
Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer
Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding.
NInfer has also improved a lot since my last post.
Now NInfer supports up to 8 concurrent requests, with a shared paged KV cache pool, each request can still use the full context length.
I aslo implemented ReplaySSM for GDN + speculative decoding, which greatly reduces the recurrent-state memory overhead under concurrency, and this is also something vllm does not yet fully supported.
There have also been many cuda kernel optimizations, as well as usage of PDL to further reduce latency.
Feedback and bug reports are always welcome, I'll try to fix issues as soon as possible!