NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5\_K\_M GGUF, q5\_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

Hardware

- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
 - Ryzen 7 7840HS, 32 GB DDR5
 - Ubuntu 26.04, nvidia driver 610.43.02 (open)

## Engine configs

****llama.cpp*\* **vLLM*\* **NInfer*\*** Quant Q5\_K\_M GGUF NVFP4 NVFP4 KV cache q8\_0 FP8 FP8 Context 196K 262K 240K MTP On (gate failed) None MTP3 (76% acceptance) Concurrency parallel=1 Continuous batch x2 lanes VRAM 31.6 GB 29.6 GB 30.5 GB

How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

- ****Relevance classification*\*** - is this industry relevant? (binary, 50 labeled deals)

- ****Needle retrieval*\*** - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context

- ****Multi-transcript QA*\*** - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls

- ****Reasoning with thinking*\*** - numeric/logic problems, thinking mode on, greedy pass@1

- ****Structured extraction*\*** - custom extraction prompt, json\_mode (skipped on NInfer, it doesn't support json\_mode)

- ****Tool replay*\*** - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache\_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

Quality results

****Tier*\* **llama.cpp*\* **vLLM*\* **NInfer*\*** Relevance 86.0% 84.0% 86.0% Needle (conditional) 100% (29/29) 100% (41/41) 100% (41/41) Transcript QA 82.0% 78.0% 88.0% Reasoning 100% 100% 98.0% Extraction F1 0.300 F1 0.350 skipped Tool replay 0% all errors 0% Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max\_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
 - Transcript QA: delta = -0.03, p=0.69 - no difference
 - Reasoning: delta = +0.02, p=0.72 - no difference
 - Relevance: McNemar p=1.0 - identical
 - Tool replay: delta = 0.0 - both fail equally

****Takeaway: quality is statistically indistinguishable across all engines.*\***

Speed results (perf probe, server-side timings)

****Metric*\* **llama.cpp*\* **NInfer*\* **Speedup*\* **Decode 1K*\* 114 tok/s 158 tok/s 1.4x **Decode 32K*\* 109 tok/s 213 tok/s 2.0x **Decode 128K*\* 72 tok/s 202 tok/s **2.8x*\* Prefill 1K 1,545 tok/s 7,265 tok/s **4.7x*\*** Prefill 32K 2,155 tok/s 6,892 tok/s 3.2x Prefill 128K 1,528 tok/s 3,904 tok/s 2.6x TTFT 1K 670 ms 138 ms 4.9x TTFT 32K 15.2 s 4.8 s 3.2x TTFT 128K 85.9 s 33.6 s 2.6x vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

What I learned