48 tg/s 440 prefill on my grandma's cluster (2xP40) (sort of)

TL;DR: switching KV cache to f16 may give a boost in speed if using MTP and ngrams.

I have a self-built "AI mega-cluster" with 2x P40s on a cheap Chinese motherboard and a Xeon CPU (around $1,100 to build, including water cooling for the GPUs). I was normally getting up to 15 tk/s with Qwen 3.8 27B Dense using a tensor split, but I suspected it was capable of much more. So, I finally asked Codex to try and squeeze out some more juice.

Disclaimer: I am not a coder at all. I’m just a generic PC user with decent overall experience, but definitely not a dev.

Recently, I’ve been running Qwen 3.8 27B Q8. With various tweaks suggested by Codex, I was able to get up to 32 tk/s on short contexts - which still falls back to an average of 12-15 tk/s on long contexts like 130K+. I was originally using Q8 for the cache (I thought it is faster because it is smaller), but then I thought, why not try the F16 cache? I did, and it turned out that F16 has much better MTP acceptance than Q8. It required fine-tuning other parameters, but it really helped improve performance.

Since I'm not good at explaining all that tech mumbo-jumbo, I asked Codex to summarize it. Sorry for the AI slop! 😄

Dual Tesla P40 / Qwen3.8-27B Q8 benchmark

Hardware: 2x Tesla P40 (24 GiB each), Xeon E5-2680 v4 (14C/28T), 64 GiB RAM. NVIDIA driver 580.173.02.

Software: llama.cpp build 5d9e5ac30 (build 10388), CUDA + locally built NCCL. Model: Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-NEO-MTP-Q8_0.gguf (28.15 GiB, 27.32B parameters reported by llama-bench).

Standard llama-bench

Three repetitions, F16 KV, all layers offloaded, tensor-parallel split over both P40s:

llama-bench -m MODEL -ngl 99 -sm tensor -ts 1/1 -dev CUDA0/CUDA1 -mg 0 \ -fa on -b 2048 -ub 512 -ctk f16 -ctv f16 -p 512,2048,8192 -n 128,512 -r 3

Test Tokens/s pp512 444.38 +/- 0.15 pp2048 432.38 +/- 0.10 pp8192 409.99 +/- 0.50 tg128 16.12 +/- 0.02 tg512 16.12 +/- 0.01 pp and tg are raw llama-bench measurements; they do not include tokenization or sampling. This tool invocation does not use speculative decoding or vision.

Actual server profile

The daily driver is a separate p40.cpp engine: F16 KV cache with one 220,160-token slot, tensor split 1:1, Flash Attention, MTP speculative decoding (draft-mtp) with ngram-simple, draft maximum 6, Qwen reasoning medium, and the F16 vision projector loaded.

Scenario Result Synthetic short 128-token decode, MTP=6 + ngram-simple up to 48.00 tok/s Typical short interactive decode observed in use (code tasks) up to 46 tok/s Long-context interactive decode observed in use about 20 tok/s 63,900-token server prefill with vision loaded 258.33 tok/s Same 63,900-token prefix, changed suffix LCP f_keep=1.000; only 4 prompt tokens recomputed in 542.53 ms The prefix result is the built-in LCP cache, not --cache-reuse. The latter is KV shifting and is disabled by llama.cpp when a multimodal projector is loaded.

For comparison, before this p40.cpp/NCCL profile the same machine was typically around 15 tok/s at long context. The roughly 20 tok/s number is an observed server result, not a llama-bench row.

Soooo, it looks like grandma GPUs still have some juice left! 😄

PS my "production" config:

-ngl all \ -sm tensor \ -ts 1,1 \ -mg 0 \ -fa on \ -c 220160 \ --fit off \ -np 1 \ -cb \ --spec-type draft-mtp,ngram-simple \ --spec-draft-n-max 6 \ --jinja \ --chat-template-file /models/qwen/chat_template.jinja \ --cache-reuse 256 \ --mmproj /models/qwen/mmproj-F16.gguf \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --reasoning on \ --reasoning-preserve \ --repeat-penalty 1.0 \ --presence-penalty 0.5 \ --min-p 0.0 \ --top-k 20 \ --top-p 0.95 \ --temp 1