How to estimate tokens/sec for your hardware

We all want more tokens per second but I keep seeing confusion on what to expect for given hardware.

For the decoding phase (TG/s) to produce one token all the model weights and KV cache needs to be read from VRAM. The compute isn't the bottleneck, only memory bandwidth.

This means we can estimate the maximum TG/s we can ever achieve given the model weights and memory bandwidth.

The math is more complicated for mixture of expert (MoE) models, but easy for dense models.

For Qwen3.8 27B Q4_K_XL, we have model weights of 16.8 GB (we exclude things not read every token; MTP layer and input embedding table)

For AMD Radeon AI PRO R9700, we have a memory bandwidth of 637 GB/s.

In the real-world it only goes down from here due to inefficiencies in the software/hardware stack. On my system running that model and hardware with llama.cpp, I get 29 TG/s, so 29 / 38 = 76% of ideal.

Also as the KV cache grows, those bytes are read for every token. Continuing the example with Qwen3.8 27B, the KV cache BF16 it costs 64 KB per token read.

VRAM GB/s TG/s = ------------------------------------------------------------ model weights GB + KV cache GB/token * context size tokens

We can make that formula more useful by moving VRAM GB/s over to the left. This allows us to plot TG/s per VRAM GS/s vs context size for a particular model.

https://preview.redd.it/9r2lw1jy9knh1.png?width=1508&format=png&auto=webp&s=6fd6ba8991c3e091ec7261e72a527478a6b89d24

1,800 * 0.0590 = 106 TG/s maximum 1,800 * 0.0293 = 53 TG/s maximum at 256k context window

- These are theorical maximums. Real-world numbers are lower due to inefficiencies in the software/hardware