Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good.
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Llama model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A user reported exceptional performance with Ninfer and a 5090 GPU, achieving up to 220 tokens per second and averaging in the 170s with a 3.8 27B model. This setup, which included ninfer-serve with parameters like --max-context 240000, --kv-capacity 240000, and --kv-dtype fp8, reportedly more than doubled throughput compared to llama.cpp. The user expressed significant satisfaction with the project's incredible job.
Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it.
If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I can't believe I was able to like double or more my throughput from llama.cpp
This is what I set up:
command: >
ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer
--model-id qwen3.8-27b-nvfp4
--host 0.0.0.0
--max-context 240000
--kv-capacity 240000
--max-concurrency 2
--kv-dtype fp8
--host-kv-mib 16384
--spec mtp --draft-tokens 3
--lm-head-draft
--vision
--media-live-mib 2048