TKintegers K5: 3.21× prefill and 3.10× decode throughput vs TorchAO FP8 on NVIDIA L4 [R] [P]
TKintegers K5, a balanced-ternary weight representation, converter, and GPU inference implementation, has demonstrated significant performance improvements over TorchAO FP8 on NVIDIA L4. It achieved 3.21x higher prefill throughput, reaching 3,402.85 tokens/s compared to FP8’s 1,061.58 tokens/s. Additionally, TKintegers K5 showed a 3.10x decode throughput advantage. The developer is seeking technical feedback on the kernel design and benchmark methodology, particularly independent replication.
This report details TKintegers K5's 3.21x prefill and 3.10x decode throughput advantage over TorchAO FP8 on NVIDIA L4, unlike other benchmarks that may not specify hardware or compare these particular quantization methods.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Oct 29, 2026, 17:53 UTC
IngestedOffset at this time: UTC+0Oct 29, 2026, 19:00 UTC
- Published
- Oct 29, 2026, 17:53
- Ingested
- Oct 29, 2026, 19:00
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
I built TKintegers, a balanced-ternary weight representation, converter, and GPU inference implementation.
The result that stood out to me: the NVIDIA L4 has native FP8 tensor-core support, but my K5 implementation achieved higher throughput than the TorchAO FP8 baseline—even though the GPU has no native K5 instructions.
On Qwen2.5-0.5B-Instruct, using a fixed matched workload:
- Prefill: K5 achieved 3,402.85 tokens/s versus FP8’s 1,061.58 — 3.21× throughput.
- Decode: K5 achieved 26.75 tokens/s versus FP8’s 8.63 — 3.10× throughput.
The baseline was standard TorchAO dynamic E4M3 FP8, running in eager PyTorch. Decode measured fixed-token model forwards with a carried KV cache.
What is K5?
TK stands for Ternary Kluster. K5 represents each weight code with five balanced ternary digits, giving 243 possible values. In this layout, each code occupies one byte. Five trits does not mean five binary bits.
My kernels expand bounded tiles for computation and use FP16 tensor-core operands with FP32 accumulation. Some tensors remain in higher precision.
For me, this is why the runtime matters alongside the representation: K5 can outperform this FP8 baseline on existing hardware without requiring a processor built specifically for K5.
These measurements apply to this model, workload, and implementation; they don’t establish a win over every FP8 runtime. The FP16 control was faster than both.
The repository includes the converter, native kernels, benchmark scripts, and recorded evidence:
https://github.com/CodeMasterCody3D/tkintegers
https://youtu.be/JFDBuwMoL0s?is=ucya9errGEh4xJ2L
I’d welcome technical feedback on the kernel design and benchmark methodology, especially independent replication.