Question: Why is prefill unbelievably faster in vLLM than other inference engines?
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
reddit.com 的 dev_community 上有用户提问，vLLM 的预填充操作为何比其他推理引擎快得令人难以置信。该用户最近开始在 4 x 48GB 4090 系统上使用 vLLM 的分支版本，并指出他们的基准测试脚本（该脚本特意清除了缓存）显示了这些令人印象深刻的结果。他们还提到，他们测量的实际缓存命中速度更快，范围在 2 万到 10 万以上。
I only started using some vLLM forks recently in a 4 x 48GB 4090 system.
DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP)
This is amazing, like having the API in my house. But it's also really hard to go back.
It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.
There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?
edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.