Prefill vs. decoding and local LLM ROI: is prefill underrated?
这条记录涉及生成能力或端侧推理进展，适合跟踪模型效率、部署门槛和应用机会。
在讨论本地运行大型语言模型（LLM）的投资回报率（ROI）时，人们通常侧重于输出速度（解码），而很少关注输入速度（预填充）。尽管解码的成本通常是预填充的3-5倍，但预填充的速度往往是解码的10-30倍，甚至在某些情况下高达50倍。例如，GLM 5.2模型在特定硬件配置下，输出速度约为60个token/秒，而预填充速度可达3000个token/秒。…
I'm trying to understand why, when people discuss the ROI of running LLMs locally, they almost always focus on output speed (decoding) and rarely on input speed (prefill), which seems like it could have a significant impact on hardware ROI.
Yesterday I saw a post on X where someone was running GLM 5.2 on 4 NVIDIA DGX Spark (4bit, speculative decoding, and other optimizations), achieving around 60 output tokens/s with 6 concurrent users in batch. Those are already great numbers. Assuming a hypothetical 24/7 agentic workload, that would be about 5.18 million output tokens per day, roughly $22/day using a price of $4.40 per million output tokens.
However, from what I read, the prefill throughput on the same setup is around 3,000 tokens/s (!)
It's true that prefill is cheaper (around $1.40 per million input tokens for GLM 5.2), but we're talking about roughly 50× higher throughput. So why does almost nobody seem to consider prefill when discussing ROI?
Even though decoding is typically 3–5× more expensive per million tokens than prefill, prefill is often 10–30× faster (and in this case, around 50× faster)... Shouldn't that have a major impact on ROI? Maybe even more than output?
Am I missing something, or is the real input/output token ratio very different from what I'm imagining?