Qwen3.6-27B - Effect of KV quantization on KLD - Q8, Q6, Q5 (bartowski)

I recently made a post here about how I squeezed more context into a Q8 model of bartowski's Qwen3.6-27B. My reasoning was that in my (anecdotal) experience, a Q8 has been performing a lot better than a Q6 or a Q5.

There were a lot of comments about quantizing KV of a higher model and some folks suggested just going with a lower quant like Q6 but with full unquantized KV. So I just wanted to test that hypothesis with KLD.

Base reference is Q8 with no KV quantization. That's because my 5090 only can fit a Q8.

- Q8 and Q6 have a steep drop the minute we put v at q4_0. Doesn't matter what quant we use for k.

- If you have to use q4_0 for v, you might as well use (q8_0, q8_0) on Q6 quant (this really surprised me)

Recommendation: Use whatever you can fit in VRAM, and just use (q8_0, q8_0). It's almost free.

I used llama-perplexity to generate this data. My primary use case for this model is only for coding and primarily python. So I wanted to use a python sample file. Downloaded a bunch of open source coding repos (transformers, torch, huggingface etc) and concatenated the python source files to generate a massive 230MB text file.

I wanted to use as high a context as my system could manage. I have a 5090 and 64GB RAM. Through trial and error, I could get up to 50K context and I just kept that for all the tests.

It seemed like the KLD improves and converges with higher number of chunks. So decided to use a chunk size of 32.

Used Qwen-3.6-27B (duh!) to put together a script to run all the different combinations. The command I used to generate the base logits was:

build/bin/llama-perplexity \ -m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \ --temp 0.6 \ --top_p 0.95 \ --top_k 20 \ --min_p 0.0 \ --repeat-penalty 1.0 \ --presence-penalty 0.0 \ -c 50000 \ -t 16 \ -ngl 99 \ --flash-attn on \ -kvo -b 1024 -ub 256 \ --kl-divergence-base ~/tmp/base_50k_coding.kld \ --chunks 32 \ -f python_corpus.txt

Once this completed, I added the additional flag --kl-divergence for the other runs to use this as the base.

Each run took 17 minutes to complete and there were 23 runs in total, so ... uh ... it took a long time.

- Learning as I go. Tell me if this is stupid or if I'm completely off base.

- As benchmarks go, I think your experience matters more. I think very often we're afraid to trust our own instinct. A benchmark isn't gospel truth.

- I don't know how important those distances are in the chart. End of the day, Q6 unquantized is 0.01 units away from Q8 unquantized. I don't know but that sounds like an insanely good compromise.

- I still want to use Q8 model. From my own personal experience, I feel it understands better and writes better code.

- I used Bartowski for no specific reason other than I have the models on my machine already. I have no opinion about Unsloth models. They may be better or worse for all I know.

model Q8_0 Q6_K_L Q5_K_L (no_kv,no_kv) 0 0.010771 0.0228 (none,q8_0) 0.005399 0.01069 0.022322 (q8_0,q8_0) 0.00541 0.010709 0.022486 (q8_0,q5_1) 0.00736 0.011715 0.023135 (none,q5_1) 0.007397 0.011648 0.023194 (none,q4_0) 0.01164 0.014789 0.024295 (q8_0,q4_0) 0.011824 0.014666 0.024101 (q4_0,q4_0) 0.020817 0.022166 0.027909