NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
This covers generation capability or on-device inference progress — worth tracking for model efficiency, deployment cost, and application openings.
NInfer now offers Day-0 support for Qwen3.8-27B, achieving approximately 200 tokens/second generation on a single RTX 5090 using speculative decoding.…
Qwen3.8-27B is finally here, and NInfer already has Day-0 support!
Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer
Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding.
NInfer has also improved a lot since my last post.
Now NInfer supports up to 8 concurrent requests, with a shared paged KV cache pool, each request can still use the full context length.
I aslo implemented ReplaySSM for GDN + speculative decoding, which greatly reduces the recurrent-state memory overhead under concurrency, and this is also something vllm does not yet fully supported.
There have also been many cuda kernel optimizations, as well as usage of PDL to further reduce latency.
Feedback and bug reports are always welcome, I'll try to fix issues as soon as possible!