50% tg increase with offloading "hot" experts to VRAM
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Llama model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A developer achieved a 50% performance boost (20 t/s to 30 t/s) in llama.cpp for Mixture-of-Experts (MoE) models like Qwen 3.8 Flash Next by offloading "hot" experts to VRAM. This optimization is particularly beneficial when the full model does not fit entirely in VRAM. The llama.cpp fork, developed by Opus, has been tested on coding workloads and is useful under specific memory constraints, though its upstream acceptance is uncertain.
I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next.
The idea is simple: instead of offloading entire layers to the GPU, I offload only the “hot” experts. I found that certain groups of experts remain relatively stable across coding, refactoring, and code-review workloads.
A couple of important caveats: this llama.cpp fork has been tested only on coding workloads, and it’s useful only when the full model cannot fit in VRAM. Will upstream ever accept it? Probably not. Opus did the low-level implementation, and I don’t feel like showing up to explain every line