ggml-cpu: use UE4M3 LUT in ARM NVFP4 dot product by ragz4125 · Pull Request #25331 · ggml-org/llama.cpp
Llama 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
此拉取请求将 #23961 中引入的 UE4M3 查找表优化扩展到 NVFP4 点积的 ARM 实现。现在，ARM 实现使用现有的 GGML_CPU_UE4M3_TO_FP32 查找表进行 UE4M3 比例解码，从而使其与 x86 实现保持一致，并重用共享的查找表基础设施。…
This PR extends the UE4M3 lookup table optimization introduced in #23961 to the ARM implementation of the NVFP4 dot product.
The ARM implementation now uses the existing GGML_CPU_UE4M3_TO_FP32 lookup table for UE4M3 scale decoding. This aligns the ARM implementation with the x86 implementation while reusing the shared lookup table infrastructure.
./build-master/bin/llama-bench -m models/Qwen3.5-4B-NVFP4.gguf -p 512 -n 0 -r 5
model size params backend threads test t/s qwen35 4B NVFP4 3.28 GiB 4.33 B CPU 4 pp512 1.89 ± 0.00
./build-lut/bin/llama-bench -m models/Qwen3.5-4B-NVFP4.gguf -p 512 -n 0 -r 5
model size params backend threads test t/s qwen35 4B NVFP4 3.28 GiB 4.33 B CPU 4 pp512 9.97 ± 0.07 The numbers shared on previous PR 23961 (ggml-cpu : add AVX2 and AVX optimization for nvfp4 dot product) looks nice.
model size params backend threads test t/s qwen3 4B NVFP4 2.63 GiB 4.02 B CPU 10 pp512 2.85 ± 0.03
model size params backend threads test t/s qwen3 4B NVFP4 2.63 GiB 4.02 B CPU 10 pp512 5.65 ± 0.03
model size params backend threads test t/s qwen3 4B NVFP4 2.63 GiB 4.02 B CPU 10 pp512 30.48 ± 0.14 Maybe it's time to try one or few tiny/small NVFP4 GGUFs on CPU-only inference.
Always happy to see CPU related optimizations so my old laptop can be more useful & efficient.