TKintegers K5: 3.21× prefill and 3.10× decode throughput vs TorchAO FP8 on NVIDIA L4 [R] [P]
TKintegers K5, a balanced-ternary weight representation, converter, and GPU inference implementation, has demonstrated significant performance improvements over TorchAO FP8 on NVIDIA L4. It achieved 3.21x higher prefill throughput, reaching 3,402.85 tokens/s compared to FP8’s 1,061.58 tokens/s. Additionally, TKintegers K5 showed a 3.10x decode throughput advantage. The developer is seeking technical feedback on the kernel design and benchmark methodology, particularly independent replication.
This report details TKintegers K5's 3.21x prefill and 3.10x decode throughput advantage over TorchAO FP8 on NVIDIA L4, unlike other benchmarks that may not specify hardware or compare these particular quantization methods.
时间与来源
时间显示为 UTC
显示时区：UTC
本地时区尚不可用，暂时显示 UTC。
发布当时偏移：UTC+02026年10月29日 17:53 UTC
收录当时偏移：UTC+02026年10月29日 19:00 UTC
- 发布
- 2026年10月29日 17:53
- 收录
- 2026年10月29日 19:00
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
档位是按信源手工设定的编辑判断，不是逐条打分。
I built TKintegers, a balanced-ternary weight representation, converter, and GPU inference implementation.
The result that stood out to me: the NVIDIA L4 has native FP8 tensor-core support, but my K5 implementation achieved higher throughput than the TorchAO FP8 baseline—even though the GPU has no native K5 instructions.
On Qwen2.5-0.5B-Instruct, using a fixed matched workload:
- Prefill: K5 achieved 3,402.85 tokens/s versus FP8’s 1,061.58 — 3.21× throughput.
- Decode: K5 achieved 26.75 tokens/s versus FP8’s 8.63 — 3.10× throughput.
The baseline was standard TorchAO dynamic E4M3 FP8, running in eager PyTorch. Decode measured fixed-token model forwards with a carried KV cache.
What is K5?
TK stands for Ternary Kluster. K5 represents each weight code with five balanced ternary digits, giving 243 possible values. In this layout, each code occupies one byte. Five trits does not mean five binary bits.
My kernels expand bounded tiles for computation and use FP16 tensor-core operands with FP32 accumulation. Some tensors remain in higher precision.
For me, this is why the runtime matters alongside the representation: K5 can outperform this FP8 baseline on existing hardware without requiring a processor built specifically for K5.
These measurements apply to this model, workload, and implementation; they don’t establish a win over every FP8 runtime. The FP16 control was faster than both.
The repository includes the converter, native kernels, benchmark scripts, and recorded evidence:
https://github.com/CodeMasterCody3D/tkintegers
https://youtu.be/JFDBuwMoL0s?is=ucya9errGEh4xJ2L
I’d welcome technical feedback on the kernel design and benchmark methodology, especially independent replication.